Cloudflare Integration With Znode

TABLE OF CONTENTS


Introduction

Znode has a Full-Page cache mechanism which often has its own limitations from the perspective of administering the full page caching on a webstore. Actions such as “clearing page-cache selectively” or “monitoring cache-related statistics” were still not possible. Additionally, the Znode’s full-page cache was driven by the same application instance. Hence, it caused the server to use its resources to serve the cached content. The requests for cached resources used to pass through the entire ASP.NET MVC Pipeline and hence the server computation-time used to occupy to some extent.

In the event of an App-Pool recycle, the application used to end up losing all its cached pages. And re-building the page cache would incur a significant cost on server resources.

The Cloudflare integration was required to rectify the majority of the above pain points. The Znode application will manage the following points of Cloudflare

  1. The Cloudflare cache purging can be done from the Znode application in 2 ways

  2. Manual Full Store Cache purge

    1. Admin users can manually purge the Cloudflare cache for the full store, See #4 here.

  3. Auto Purge Specific URL Cache

    1. The specific cache will be auto-purged from Cloudflare when you will publish the below-mentioned elements:

      1. Store

      2. Catalog

      3. Category

      4. Product and 

      5. CMS


Assumption: 

  1. Users should have at least a business account on Cloudflare.

  2. This document doesn’t cover detailed Cloudflare knowledge and assumes that the administrator has a fair knowledge of Cloudflare administration.

  3. The Znode application should be hosted on a public IP/domain so that Cloudflare can access the Znode application.


How to use Cloudflare caching feature with Znode

By doing the following steps Cloudflare will cache the Znode webstore application

Setup needs to be done in the Cloudflare portal

  1. Add the website URLs in the DNS tab with Name and IPv4 address. This setup will host the website on Cloudflare.Note - If the website is already hosted then this step can be skipped

Note - If the website is already hosted then this step can be skipped

  1. Add the following rules in the “Page Rule” tab of the Cloudflare account to bypass non-cacheable or dynamic content URLs (which needs real-time values rather than cached content)

    1. Rule 1 - Bypass GetCartCount Ajax

      1. URL - {domain name}/Home/GetCartCount

      2. Setting - Cache Level: Bypass

    2. Rule 2 - Bypass Login Ajax

      1. URL - {webstore URL}/User/Login*

      2. Setting - Cache Level: Bypass

    3. Rule 3 - Bypass Cart

      1. URL - {webstore URL}/cart*

      2. Setting - Cache Level: Bypass

    4. Rule 5 - Bypass recent view products

      1. URL - {webstore URL}/Product/GetRecentViewProducts*

      2. Setting - Cache Level: Bypass

    5. Rule 6 -  Bypass checkout page

      1. URL - {webstore URL}/checkout/index*

      2. Setting - Cache Level: Bypass

    6. Rule 7 -  Bypass order receipt

      1. URL - {webstore URL}/Checkout/OrderCheckoutReceipt*

      2. Setting - Cache Level: Bypass

    7. Rule 8 -  Bypass wishlist page

      1. URL - {webstore URL}/User/Wishlist*

      2. Setting - Cache Level: Bypass

    8. Rule 9 -  Bypass user history page

      1. URL - {webstore URL}/User/History*

      2. Setting - Cache Level: Bypass

    9. Rule 10 -  Bypass user address URL

      1. URL - {webstore URL}/User/AddressBook*

      2. Setting - Cache Level: Bypass

    10. Rule 11 -  Bypass user review URL

      1. URL - {webstore URL}/User/Reviews*

      2. Setting - Cache Level: Bypass

    11. Rule 12 -  Bypass gift card history URL

      1. URL - {webstore URL}/User/GiftCardHistory*

      2. Setting - Cache Level: Bypass

    12. Rule 13 -  Bypass profile page

      1. URL - {webstore URL}/User/EditProfile*

      2. Setting - Cache Level: Bypass

    13. Rule 14 -  Bypass saved credit card URL

      1. URL - {webstore URL}/User/GetSavedCreditCards*

      2. Setting - Cache Level: Bypass

    14. Rule 15 -  Bypass affiliate information page

      1. URL - {webstore URL}/User/AffiliateInformation*

      2. Setting - Cache Level: Bypass

    15. Rule 17 - Bypass GetPriceWithInventory Ajax call

      1. URL - {webstore URL}/Product/GetPriceWithInventory*

      2. Setting - Cache Level: Bypass

    16. Rule 18 - Bypass LoginStatus Ajax call

      1. URL - {webstore URL}/User/LoginStatus*

      2. Setting - Cache Level: Bypass

    17. Follow the same steps to add additional URLs which you would like to serve dynamically without cache. The same goes for any custom Page or Ajax request which you would like to bypass.

  2. Add the following rules in the “Page Rule” tab of the Cloudflare account to cache everything. This rule should be added to the last of the rules list. 

    1. Cache everything of webstore URL

      1. URL - {webstore URL}/*

      2. Setting - Cache Level: Cache Everything, Edge Cache TTL: 12 hours

      3. Order - Last

        Note - An in-depth documentation about setting up page rules for a website can be found on the Cloudflare Official page - https://support.cloudflare.com/hc/en-us/articles/218411427-Understanding-and-Configuring-Cloudflare-Page-Rules-Page-Rules-Tutorial-#overview
  1. Users can prevent web scraping/High precision DOS attacks. The same has been described in the link https://www.cloudflare.com/learning/bots/what-is-rate-limiting/

  2. Cloudflare can restrict URL access to trusted IP ranges using Zone Lockdown. https://support.cloudflare.com/hc/en-us/articles/115001595131-Understanding-Cloudflare-Zone-Lockdown

Sample instruction:

  • Ask customers for which regions; Mexico, CA, UK.

  • Ask customers for which 3rd party apps are going to need access to the API.

  1. In Cloudflare, different firewall rules can be set for e.g. Blocking production site access for countries or continents https://developers.cloudflare.com/firewall/cf-firewall-rules/

  2. Important: By default, Cloudflare caches the images. Znode serves images from the API domain, so there is no need for any special rule to cache images coming from APIs (if the API domain is already proxied through Cloudflare). In fact, this was observed that, if a rule is put to bypass the API domain from the cache it would negatively affect the caching of images as well.


Setup needs to do in the Znode API Config file

There are few keys related to Cloudflare, which need to be set up in API’s web. config file:

  1. In API’s web.config file, Under “CloudflareConfigSection” change the value of “CloudflareApiKey”. Where to get the value of this key in Cloudflare?



Setup needs to do in the Znode admin application

  1. Enable the Cloudflare from the following menu

    1. Store -> Additional Attributes -> Cloudflare Setting

  2. Add zone id provided by Cloudflare account. The zone id can be found in the overview tab of the Cloudflare account.


Purging of Cloudflare cache 

From Znode application

  1. The Cloudflare purging from the Znode admin application can only work if the Enable Cloudflare flag is enabled for the store.

  2. When an admin user will publish the Store, Catalog and CMS content from the Znode admin application then all the Cloudflare cache will be cleared of all the websites which are associated with provided zoneId to the store.

  3. When an admin user will publish the category and product from the Znode admin application then the cache of URL associated with category or product will be purged.

  4. Cloudflare caching can be purged manually from Admin -> Global Setting -> Cache management. Here users can select the domain for which Cloudflare cache needs to be purged.




From Cloudflare account

Purging of Cloudflare cache can be done from Cloudflare account in two ways

  1. Purge everything - All Cloudflare cache can be cleared from the Purge everything button in the caching tab of the Cloudflare account.

  2. Custom Purge - Custom purge provides three options to purge Cloudflare cache

    1. Purge by URL - Any cache related to a particular URL will be purged from the Cloudflare cache.

    2. Follow this link for some Cloudflare enterprise account features around cache purging. https://support.cloudflare.com/hc/en-us/articles/200169246-How-do-I-purge-my-cache-#2Cmoyau5BdK0sEYFlpoxq8

Common Issues and their workaround:

  1. Znode Webstore and Admin Performance issue

    1. Issue - When the API is added to the project's DNS entry in CloudFlare (As Proxied) (Screenshot below) to serve the DAM images through CF CDN, it starts to impact the Webstore and Admin app performance.


    2. Root Cause - The Webstore and Admin app perform slow, because, to reach the API server, both the apps will have to go through the CF proxy server over an internet connection. (See diagram below)


    3. Workaround

      1. Preferred option - This option provides the best performance.“vhost” file can be configured on the Windows machine of the Webstore/Admin server, which helps in DNS lookup. If the API is on the local machine, an entry can be made such that the API domain resolves to localhost. If the API is not on the local machine, an entry can be made such that the API domain resolves to the IP of the machine hosting the API, or, to the IP of the load balancer in front of the API nodes if applicable.
        In this “vhost” file, the API “domain” can be mapped with a private network “IP” (See diagram below)


      2. Alternative option - This option increases latency and should be avoided, but is noted here for the sake of documentation. Keeping the CF Proxy server and Znode server in the closest possible proximity to have minimal latency can be tried. In this approach, the hosting server doesn’t require any additional configuration/handling. Which will be the desired option in multi-node (LB) deployment. But still, it will have the “internet” network cost, and won’t have the most optimum performance. And also, CF CDN is an edge service, so it can’t really be achieved always.

  2.  Znode Publish or Search Index creation process:
    Note: This is also related to the above issue #1, but specifically occurs with publishing and with any long running HTTP Requests while working with CloudFlare.

    1. Issue - It has been observed that when Znode API is proxied (DNS configuration) through Cloudflare, the Znode Search Index process was failing for a large (time-consuming) catalog. It used to timeout the connection. And in Znode, the “Index failed” message is seen as there were not ample logs in the TaskScheduler>Indexer. cs class.

    2. RCA - Below is the “Publish / Create Index” request flow, with an explanation:

      The call between TaskScheduler and Indexing API was connected through CloudFlare proxy. And as it is known, Indexing process may take up to many minutes based on catalog size/network, and Cloudflare will not respect the "timeout" setting that are applied on WebRequest in TaskScheduler>Indexer. cs class and it terminates the request if it takes more than 100 seconds. Which will result in Index process failure.
      Upon doing some more debugging and logs, it was seen that the TaskScheduler was receiving the “Connection Closed” error with Error 524 code.
      This “Error 524” code is Cloudflare specific code, and after searching for it, it was found that in CloudFlare, “Any HTTP request that doesn't receive a response from the origin server for over 100 seconds (or over 600 seconds for enterprise customers) will be timed out and you'll see the 524 A Timeout Occurred error.”

    3. Workaround - To avoid TaskScheduler to connect to Znode APIs via CloudFlare proxy, API's DNS lookup was added on the local Virtual Host (Vhost file) file in windows.

      The API domain was added with the corresponding Private IP in the Vhost file. So now, DNS for API will be looked up locally, and it will connect to API on the private network, rather than routing through CloudFlare.


Limitations:

  1. Only web stores are compatible to work with Cloudflare caching and page rules defined above.
  2. Cloudflare does not provide any facility which allows caching for a particular section on the web page.
  3. Real-time ajax request URLs need to be specified under Cloudflare page rules to exclude them from caching. This takes some time to get the perfect setup for any site.


Important:

  1. Cloudflare is enabled for Webstore only.
  2. Real-time data of price and inventory will be displayed on PDP using Ajax calls and data on PLP will always come from the Cloudflare cache.
  3. After purging the Cloudflare cache, users will have to wait for 30 sec to see an impact on the web store.
  4. If the setup of caching has been done in the Cloudflare account but not enabled from the Znode admin application then Cloudflare cache purging will not work from the Znode application.


Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.