SiteMapGenerator

GRAILS microservice that generates the XML sitemaps for the Westend61 website. The data is read live from ElasticSearch, nothing is stored in a database.

• GRAILS 6.2.3, Groovy 3.0.23, Gradle 7.6.4
• JDK 17 (the build fails with any other Java version)
• ElasticSearch java client 8.x
• only minimal jars included (no hibernate/GORM, no database; Quartz needs a DataSource bean, so an in-memory H2 is configured, it is never used)

Sitemaps

Sitemap	Content	Languages	ElasticSearch index
images	image previews (image: sitemap extension)	en, de, es, it, fr	w61assets
videos	video previews (video: sitemap extension)	en, de, es, it, fr	w61assets
categories	category pages built from relevant user searches	en, de, es	w61searchlog

Endpoints

All URLs are available per language (/{lang}/...), the app listens on port 9099.

URL	Description
/{lang}/imagesSitemap/sitemap.xml	index sitemap, listing every page sitemap of the images
/{lang}/imagesSitemap/sitemap-{n}.xml	page n of the images, 36000 per page, page 1 = newest images
/{lang}/imagesSitemap/init	rebuild the page cache and start the SitemapCacheJob for the images
/{lang}/videosSitemap/sitemap.xml	index sitemap, listing every page sitemap of the videos
/{lang}/videosSitemap/sitemap-{n}.xml	page n of the videos, 36000 per page, page 1 = newest videos
/{lang}/videosSitemap/init	rebuild the page cache and start the SitemapCacheJob for the videos
/{lang}/categoriesSitemap/sitemap.xml	one sitemap with the category links (en, de, es only)
/{lang}/categoriesSitemap/test	plain text list of the searches used for the categories (en, de, es)
/, /de/sitemap, /en/sitemap	simple HTML page with links to all sitemaps

Examples:

• http://localhost:9099/en/imagesSitemap/sitemap.xml
• http://localhost:9099/de/imagesSitemap/sitemap-1.xml
• http://localhost:9099/de/videosSitemap/sitemap.xml
• http://localhost:9099/de/categoriesSitemap/sitemap.xml

Add ?noid to a video page URL to get only the video URLs without the additional data. The image sitemaps always contain only the page and image URLs (Google no longer uses image:title, image:caption and image:license).

How it works

• images / videos: the page boundaries are the smallest resourceId of each page. This page cache is built on the first request (or by calling init) with one ElasticSearch query per page. Every following page request only needs one query. Call init after new assets have been indexed, see tools/running/reinitSitemaps.sh.
• pre-rendered pages (images / videos): the Quartz job SitemapCacheJob (group sitemaps) renders every page in every language (en, de, es) into gzipped files below sitemap.cache.dir. It is triggered 20 s after the start and every night at 03:30, and is started by init (force). Without force a type is skipped while its complete cache is younger than sitemap.cache.minAgeMinutes (1 week, minus 1 hour tolerance), so a cache is rebuilt about once a week. A restart keeps the cache if sitemap.cache.dir is kept. The pages and the page boundaries are written as one generation (<dir>/<images|videos>/gen-<yyyyMMddHH>/, the UTC hour of the start, -2, -3 ... if there are more in the same hour; the exact start time is created in its manifest) and published at once by switching the CURRENT file, so a stopped or failed run never changes what is served. The previous generation is kept for requests still reading it.
  • The controllers send the file as it is (Content-Encoding: gzip) or unzip it for clients without gzip, with ETag, Last-Modified, Cache-Control: public, max-age=... and 304 for conditional requests.
  • The <lastmod> of the index only changes when the content of a page has changed (SHA-256 of the XML).
  • If a cached page exists it is always delivered, also for the video ?noid variant (the additional lines are removed while streaming, with its own ETag). Add ?force=1 to a page or index URL to ignore the cache and create it live from ElasticSearch.
  • A page that is not in the cache is created, stored in the cache and then sent from the stored file, so the first and every later request deliver the same bytes. Only one request creates a page, concurrent requests for it wait. The cache always stores the complete page, also if the miss was a video ?noid request. ?force=1 does not store. If the page can not be written (disk full ...) it is sent live without the cache.
  • If there is no generation yet, the first miss creates an incomplete one (complete=false in its manifest) with only the requested pages and the current page boundaries. The job does not count it as fresh and replaces it with a complete generation. Until then the index is created live. init never deletes cached pages: they are served until the job has published the complete generation, which then replaces them (the previous generation is kept for requests still reading it).
  • A full run needs about 30 s per page and language, because of the 36000 assets per page. During this time the old generation is served. Deploying without keeping sitemap.cache.dir means the first run starts from scratch.
• categories: uses the searches logged in w61searchlog with at least 500 results and 50 uses (25 uses for es), sorted by use count. Searches containing bad words are removed (SoapService).
• The host used for the URLs in the sitemaps is grails.serverURL of the current environment.

Configuration

grails-app/conf/application.yml:

Property	Default	Description
elasticsearch.host	localhost	ElasticSearch host
elasticsearch.port	9200	ElasticSearch HTTP port
elasticsearch.cluster.name	search3stage	search3stage for development/stage, search3 for production
elasticsearch.socketTimeoutSeconds	300	read timeout of the ElasticSearch client, its default of 30 s is too short for a sitemap page
sitemap.cache.enabled	true	serve the pre-rendered pages, false creates every page live
sitemap.cache.dir	./sitemap-cache	directory of the pre-rendered pages (env SITEMAP_CACHE_DIR also works)
sitemap.cache.maxAgeSeconds	3600	Cache-Control: max-age of the pages
sitemap.cache.minAgeMinutes	10080 (1 week)	the scheduled job skips complete caches younger than this
grails.serverURL	per env	http://localhost:9099, https://www.stage.westend61.com, https://www.westend61.de

Build & run

sdk env                    # JDK 17, GRAILS and Gradle versions from .sdkmanrc
./gradlew bootRun          # development environment, port 9099
./gradlew assemble         # creates build/libs/SiteMapGenerator-<version>.war

• The project jars come from our Artifactory. Add the password to your user gradle properties (~/.gradle/gradle.properties), never to the project:

  artifactoryPwd=...
  

  (CI can use the environment variable ORG_GRADLE_PROJECT_artifactoryPwd instead.)
• For local development ElasticSearch must be reachable on localhost:9200. Use an SSH tunnel: ./PortForwardStage.sh (stage) or ./PortForwardPROD.sh (production).

Deploy

Script	Target
./deploy2STAGE.sh	stage, www.stage:/home/sitemap/
./deploy.sh	production, www.westend61.de:/home/sitemap/

Both build the WAR with the matching -Dgrails.env and copy it with rsync to the temporary folder deploy/ of the server. The scripts and docker/docker-compose.yml go to /home/sitemap/ itself. Then stopDeployStart.sh is run on the server, it runs the app as docker container (docker-compose.yml):

1. stops the container (docker compose down)
2. moves SiteMapGenerator.log to LOGBACKUPS/ (if this folder exists) or empties it
3. copies the new WAR from deploy/ to /home/sitemap/SiteMapGenerator.war
4. starts the container and waits until its health check is healthy (START_TIMEOUT, default 300 s). The health check is healthcheck.sh: it requests the start page and the index sitemaps (images, videos; en, de) and checks for HTTP 200, an XML content type and a complete <sitemapindex> with entries
5. runs the smoke test (smoke-test.sh), its result is the exit code

The files in /home/sitemap/:

File	Description
docker-compose.yml	the container, ElasticSearch is reached on localhost:9200 (host network)
SiteMapGenerator.war	the WAR in use
SiteMapGenerator.log	the log of the app, written from inside the container (and shown by docker compose logs)
sitemap-cache/	the sitemap cache
deploy/	temporary, only the place where the deploy scripts put the new WAR

The whole folder /home/sitemap is mounted as /app into the container, so nothing but the WAR has to exist before the start. The container runs as the owner of the folder (APP_UID, APP_GID), the app listens on 127.0.0.1:9099 only. The first time the old service has to be stopped (./SiteMapGenerator.war stop), it uses the same port.

Options: --no-clean|-nc, --no-deploy|-nd, --no-stop|-ns, --only-tools|-ot (only copy the scripts and the docker-compose.yml), --help|-h.

Tools

File	Description
tools/nginx.conf	nginx locations for the sitemaps
tools/running/reinitSitemaps.sh	rebuild the page cache for images and videos
tools/running/stopDeployStart.sh	stop the container, take the new WAR from deploy/, start it, smoke test
tools/running/cleanSitemapCache.sh	remove old generations and temp files of the sitemap cache (-n = dry run, -a = min. age in hours)
tools/downloadAllSitemaps.sh	download all sitemap pages (to check them)
tools/test-elastic.groovy	test the ElasticSearch connection
es-scripts/	queries to find the relevant searches for the categories

nginx config

/etc/nginx/conf.d/sitemaps.inc, the complete file is tools/nginx.conf (images, videos and categories in en, de, es):

location /en/imagesSitemap/ {
        proxy_pass http://127.0.0.1:9099/en/imagesSitemap/;
}
location /de/imagesSitemap/ {
        proxy_pass http://127.0.0.1:9099/de/imagesSitemap/;
}
...

robots.txt

Sitemap: https://www.westend61.de/en/sitemap/sitemap.xml
Sitemap: https://www.westend61.de/de/sitemap/sitemap.xml
Sitemap: https://www.westend61.de/en/imagesSitemap/sitemap.xml
Sitemap: https://www.westend61.de/de/imagesSitemap/sitemap.xml
Sitemap: https://www.westend61.de/en/videosSitemap/sitemap.xml
Sitemap: https://www.westend61.de/de/videosSitemap/sitemap.xml
Sitemap: https://www.westend61.de/en/categoriesSitemap/sitemap.xml
Sitemap: https://www.westend61.de/de/categoriesSitemap/sitemap.xml

The first two sitemaps (/{lang}/sitemap/sitemap.xml) have no mapping in this app, they must come from somewhere else.