All Collections
Member Management
Member Directory
Deep Dive: Data Harvesting & Website Scraping
Deep Dive: Data Harvesting & Website Scraping

Learn how bad actors scrape data from membership directories and event registration lists to sell back to your members and how to fight it.

Pete Zimek, CAE avatar
Written by Pete Zimek, CAE
Updated over a week ago

Bad actors have been mining websites for data since nearly the founding of the Internet. In most cases, these individuals are collecting this data to be sold back to marketers, and they are using the very lists they're selling to market their own goods. The end result is lots of spam that leads to even more spam. Of course, all of this is done without your permission and without the permission of your members.

Hear from your peers:

In this article...


Harvesting Methods

Harvesting can be done by hand, with a person manually going member by member to copy & paste email addresses into a list.
โ€‹
More sophisticated spammers use bots, essentially computer programs, to crawl page by page. These bots can be incredibly accurate and lightning fast.


Associations Are Targets

For many of these bad actors, an association's public-facing member directory and event registration lists can be a target-rich environment with hundreds or even thousands of up-to-date email addresses ready to be harvested. And association staff knows perhaps better than most, that there are a lot of companies that would love to get their hands on that data for their own questionable marketing practices.

Even ASAE Isn't Immune

The American Society of Association Executives (ASAE) is known as one of the top thought-leading organizations in the association space, and their technology team is known for being one of the best in the industry. The screenshot below taken on 6/16/2018 from my email account is clear evidence that ASAE members, attendees, and exhibitors are in the crosshairs of spammers too.

You'll notice that I received six different emails attempting to sell me the list of attendees for ASAE's 2018 meeting. The spammers likely found my address by scraping ASAE's own website. As a member of a number of associations and someone who attends trade shows and conferences throughout the year, I have been receiving messages like these for almost as long as I have had a professional email account.



Web Scraping is an "Internet Problem"

The one overarching takeaway is that this isn't an association problem or a Novi AMS problem. It's an Internet problem. The second we publish data to the Internet, literally the entire world can see it.

While association staff put tremendous pride in the customer service provided to members, on this issue, I think that it's important to set clear expectations that much of this is outside of your control and ours as well.

Just like every email account will eventually receive spam, every member directory will eventually be abused. That is a fact of this digital space that cannot be changed.


Fighting Data Harvesters

There are a number of actions that we can take to combat these bad actors.

Mark emails as spam

These actors are overseas, so a threat of legal action might only provoke them. My personal approach is to mark the incoming messages as spam in Outlook. Outlook and Gmail use sophisticated artificial intelligence to monitor the way you interact with messages in your inbox to determine how (and if) they will deliver similar messages in the future. If the spammers can't sell your data, they will be less incentivized to steal it.

If enough of us mark these messages as spam, I am convinced that we can put a dent in this problem. The best part... this is an action that all of us, association staff, the Novi team, and even your members, can take.

Lockdown the member directory

Perhaps the fastest way to protect your member data is to lock your member directory down. For many associations, a freely available member directory is considered an important member benefit for the members listed in the directory, so locking the entire directory down to members-only is a business decision that might not be practical. However, you can also choose to show contact info to the public or not, still highlighting the records without providing their information to those without access.

Lockdown email addresses

For those of you who decide that the benefits of an open directory outweigh the risks, we have put in place our own layer of protection from bots. This article explains how we use reCAPTCHA to force anonymous website visitors to prove that they're human prior to showing them a member's email address. Keep in mind that this does not stop real people from harvesting your data.

Use a contact form instead of showing emails

Associations also have the option of using our member-to-member contact form. This removes email addresses from the member directory and replaces them with a form to enter their Email Address, Subject, and Message. Unfortunately, this is not a silver bullet solution either, but at least gives another option.

Limit the number of emails a user can send via the contact form

If the contact form is enabled, you can also use the Contact Form Limit setting to enforce a maximum number of messages that can be sent to your members from the same user within a specific time period. This setting can be accessed in your Association Settings (Membership tab).

Use the Email Blocklist to block a specific sender

If someone is sending repeated messages through the contact forms on your website, Association Admins can use the Email Blocklist to block the sender from sending additional messages.

Alert your members

Consider adding a message to your website that lets members know what to expect when it comes to messages they receive from your site. See the screenshot below for an example that is set up in the Emergency Alert.


Related Articles

Did this answer your question?