Monitoring password-protected websites using Visualping is easy and straightforward using its Perform actions feature. Here's how to do it.

There are various tools that can help you avoid relying on the refresh button when you want to monitor webpages for changes in, for instance, appointment openings, out of stock products and stock prices.

But few services allow you to monitor webpages behind a password-protected site. Visualping is one of the easiest tools to track changes to webpages barracaded behind the brick wall of a login form.

There are more password-protected pages than you'd think users need to track. To name a few, the U.S. Naval Academy Alumni Association (to track new events and newsletters), Jira Software (to track new issues), Password-protected Atom or RSS feeds (to track new feed items), and various sites that specialize in government request for proposals (RFPs).

Read on to master the life hack of monitoring password-protected sites.

Tracking a Webpage Behind a Login Form

Method 1: Using the Visualping Chrome Extension

Simply install the Visualping chrome extension, log in to the app you want to monitor, and create the job using the extension. If you select Server mode, all cookies will be captured as if you had done it manually (see Method 3). If you select Device mode, your session will be used to monitor the page.

Method 2: Employing the Element Selector

To monitor a password-protected webpage with Visualping, you can use our element selector to instruct the crawler to type your login information into the username/email and password forms. Then you use the element selector again to click the Enter button (or Submit, Login In, etc) to instruct the crawler to login before it takes a screenshot of the webpage.

Step 1: Head over to Visualping in your web browser. You don’t need to first sign up.

Step 2: Copy and paste the URL of the password-protected page into the search bar and press GO. For example, you can try with the dummy form below:

Step 3: The Advanced section of Visualping will automatically appear. But, to see more options, click the downwards arrow beside Perform actions.

Step 4: Under Click, select Type in the dropdown options. Type your username or email - whichever you normally use to access the password-protected page - in the field that reads "Words to type". Then click the arrow off to the right - the element selector - and use it to click the webpage element of the username/email form, up above in the screenshot area.

By clicking the username/email form with the element selector, you are instructing the crawler to type your username/email information into that form when it goes to check for webpage changes.

But if you're unable to click the correct element using the element selector, you can also use the inspect function in any modern web browser to find the ID of the webpage element you wish the crawler to type in. Copy and paste the element ID in the field that reads "Field name (or XPath)", and enter your email in the "Words to type" field. The system will then crawl the page for a “typable” field with the ID you entered.

Another option is to use the XPath. Type in “//*[@id="email"]” in the field that reads "Field name (or XPath)", and enter your email in the "Words to type" field. The system will then crawl the page for a "typable" field with the XPath you entered.

Step 5: Next, click + Add action to add another action for the crawler. Under Click, select Type in the dropdown options. This time, type your password in the field that reads "Words to type". Then, in order to instuct the crawler to type your password into the password form, select the element selector, and use it to click the webpage element of the password form above in the screenshot area.

Again, you can also use the inspect function instead of the element selector. Simply copy and paste the element ID into the "Field name (or XPath)" form, and type in your password in the "Words to type" form.

Or, type the XPath (//*[@id="password"]) under "Field name (or XPath)", and your account's password in "words to type". The system will then crawl the page for a "typable" field with the ID or XPath you entered.

Step 6: The last step is to instruct the crawler to click Enter (or Submit, Log In, etc) before it takes the screenshot of the password-protected site. Select + Add action one last time. But, under Click, select Click in the dropdown options. Use the element selector to click the Enter button.

But, if you're unable to click the correct element using the element selector, you can just type "Enter" (or "Submit", "Login In", etc) in the "Button name (or XPath)" field. Or, after selecting Click in the dropdown options, type the XPath name “//*[@id="enter"]” in the "Button name (or XPath)" field.

Step 7: Depending on how long it takes for the page behind the login form to appear, you might need to instruct the crawler to wait a few seconds before it takes the screenshot. Click + Add action (just one more time - we promise!), select Wait in the dropdown options under Click and type “2”, or more, in the "Number of seconds" field. Click GO to confirm it works.

Step 8: Once you see the resulting page in the screenshot area, you can finish setting up your track as you normally would. Choose the frequency of checks -every 5 minutes, 30 minutes, 3 hours or once daily, etc.

Step 9: Enter your email address you wish to receive email alerts from under Send notifications to.

Step 10: And finally, click START FREE MONITORING.

You may be asked to confirm your email by Visualping if this is your first time using the service. If this is the case, you will need to check your mailbox to find an activation email. Once you click the link in the email, you can set a password for your new account at Visualping. Finally, you’ll be taken to your dashboard wherein you can create new jobs, check and configure active as well as inactive jobs, check history of jobs, check account settings, and do a lot more.

Method 3: Using your cookies manually

Cookies are the text information saved by a website in your web browser. It was invented to save session information on your computer so that the website can recognize users, i.e., you don’t have to enter the password again when you’re returning to a website. Without cookies, you would need to login to a website every time you visit it. Also, cookies store a lot of other information too, but will not be relevant to this post.

The cookies storing login data are called session cookies, and they exist for a limited amount of time known as “open session”. They can exist for a short time (say, five minutes at most) for a banking application or for a long time (say, a month) for less critical sites. Since a website stores a lot of cookies for various purposes, this method is relatively harder than the above method as there is some trial and error involved in identifying what is the right cookie to use. The cookie method, however, is the only way to monitor a page subject to a two factor authentication (this will be the subject for another post).

First of all, you must know how to extract cookies from a web browser:

Step 1: Open Chrome in Incognito or Firefox in Private Browsing mode.

Step 2: Open your target website and complete the login process.

Step 3: Right-click on the page, click on Inspect or Inspect Element to access the developer tools. In there, click on the Application or Storage tab.

Step 4: You can sort the cookies by Expires or Size and pick the cookies with an expiration of two weeks to months or larger sizes to be the important ones. If you are in doubt, use third-party services like CookieServe to understand them. If all fails, you can pick all the cookies and enter their details in Visualping.

Now, you must follow Method 1 till Step 3, then continue as given below:

Step 4: Click Cookie under Click (the dropdown), and enter the cookie details. You can press GO to check if the login is getting bypassed using this cookie.

Step 5: If you need to enter more cookies, click + Add Action and follow Step 4.

Finally, follow Method 1 from Step 7 till you set up Visualping successfully.

How to find the XPath, Selector or element IDs

XPath is a query language for selecting elements in XML language, which also works for the HTML language used in web pages. In short, it helps to select elements on a page, which further helps to do some operation on the element.

Which is the right element to choose in the hierarchy? The best element to choose is the input element (which is highlighted in the screenshot). When you first choose Inspect for a hovered element, you might not see it highlighted under Elements or Inspector. If not, then expand the element until you see the input element. Also, when you hover or highlight an element under Elements or Inspector, you can see it highlighted on the web page as well. You can use this trick to confirm you’re selecting the right element. For example, if you’re looking to select the Email field, check that the same field is highlighted on the page.

Using Google Chrome

If you’re using Chrome, just hover over the element you wish to inspect, then right-click, and choose Inspect from the context menu. In the Elements tab of the Developer Tools, the element will be highlighted; right-click over it, then go to Copy, and finally choose Copy XPath from the given options.

This trick will work on all Chromium-based browsers including but not limited to Brave, Opera, Vivaldi, and the new Chromium-based Microsoft Edge.

Using Mozilla Firefox

If you use Firefox, simply hover over the element you need to inspect, then right-click, and select Inspect Element from the shown menu. In the Inspector tab of the Developer Tools, you’ll see the element is highlighted; just right-click the element, then go to Copy, and finally click on XPath.