Also, there was an issue where it was impossible to create paths for hierarchical pages as there wasn’t enough information at hand to generate the path in those situations.I toyed with the idea of looking at the front end of the site (i.e. not the API) to find the URLs for each page, but not only was it difficult to generate the Jadu paths, it was also impossible to confirm that I had the correct URL.Getting to this point took quite a bit of trial and error. It quickly became apparent that generating the Jadu paths in the Drupal site was not going to work and that simply collecting together all of the information we had at hand was the best solution.I will release the source code for this migration in the coming weeks. As I write these articles I am tidying things up so that the module can be released as a self contained Jadu to LGD migration starter module. There is some custom logic for Central Bedfordshire that might not be useful for everyone so I’m working on either removing that or creating configuration to allow those customisations to be configured.For example, let’s say we had a page with the path /news/article/1473/housing_matters_-_january_2026 for a news article. I could attempt to generate the path using the title of the page and Jadu’s rules on creating URL slugs. Unfortunately, if I attempt to access the same page with the completely incorrect URL /news/article/1473/monkey-monkey-monkey it results in a 200 status code with the correct content. The page itself doesn’t have a canonical URL metatag so there is no way to verify that the URL I attempted to fetch is “correct” using the site.Since I was migrating into a LGD site, it made sense to use the Drupal path auto system and LGD path management plugins to manage the paths on the Drupal site. We therefore needed to know the existing Jadu URLs so that we could create these redirects.The following class is the even subscriber for the Jadu migrate module that will fire the onMigratePostRowSave method when the MigrateEvents::POST_ROW_SAVE event is triggered. Using this event we know that the new entity is in place and so we can add extra items to the database to fill in any gaps in the data (like the redirects for example).
What I Tried
The site had a sitemap.xml file that I could use, although it was clearly corrupt as the XML wasn’t valid I could at least get lots of URLs from the file. In the run up to the migration the client had been collecting URLs from the site via SEO sites scanning tools so this was able to fill in some of the gaps that existed in the sitemap.xml file. Outside of that I also had a number of URLs that didn’t appear to exist anywhere else, but were valid pages on the site (found whilst running the migration and comparing the new paths with the imported redirects).With everything working together here we had a good mechanism for both “generating” Jadu URLs and for creating redirects between one site and the other.<?php
declare(strict_types=1);
namespace Drupaljadu_migrateService;
use DrupalredirectEntityRedirect;
use DrupalredirectRedirectRepository;
/**
* Create a path alias based add a redirect from the old URL where necessary.
*/
class AliasRedirect implements AliasRedirectInterface {
public function __construct(protected RedirectRepository $redirectRepository) {}
public function redirectExists($redirectSource):bool {
$redirects = $this->redirectRepository->findBySourcePath($redirectSource);
if (count($redirects) > 0) {
return TRUE;
}
return FALSE;
}
public function createRedirect(string $type, int $oldId, int $newId, ?int $parentId = NULL):void {
switch ($type) {
case 'localgov_news_article':
$jaduPath = JaduUrls::findUrlById((string) $oldId, $type);
break;
default:
// No redirect logic for this type.
return;
}
if ($jaduPath === NULL) {
// Don't create a redirect for a path we don't have.
return;
}
if (str_starts_with($jaduPath, '/')) {
// Remove the first slash if present.
$jaduPath = substr($jaduPath, 1);
}
if ($this->redirectExists($jaduPath) === TRUE) {
// Redirect already exists.
return;
}
$redirect = Redirect::create([
'redirect_source' => $jaduPath,
'redirect_redirect' => 'internal:/node/' . $newId,
'language' => 'und',
'status_code' => '301',
]);
$redirect->save();
}
}
If you migrate a page from one system to another then it is highly important that you maintain the URL structure of the site. If you change the URL of a page then you need to add in a step that adds a redirect from the old system to the new so that all of your search engine results, the existing links from other sites, and any user bookmarks that have been created work correctly. This is critical to get right for a public facing council site like this.
- Replacing spaces with dashes
- Removing stop words from the title
- Changed to lowercase
- Special characters removed
In the end this was around 48K URLs, which included links to images and other files in use on the site.To get the URLs during the migration caused quite a bit of experimentation, but I did solve the issue with a solution that had a high success rate.As this system wasn’t perfect we did spot a few missing URLs. Creating a report to show gaps was important as it allowed us to track down a couple of edge cases where the logic wasn’t working and where we didn’t have records of the URLs in question. Even then there were some redirects that needed adding in manually and we are keeping an eye on the 404 responses on the site to catch any other missing pages that might have crept in.It became clear that what I needed was a way to find the URL of a page from a known list of URLs, rather than trying to reconstruct or attempt to pull them from the site.Also, because the Jadu API is rate limited you need to slow down how often you access the site of you will be banned. To bypass this rate limit I added an artificial delay of 3 seconds to all of the API requests in the migration, which was an acceptable delay in the migration. Adding more API calls would have slowed down the migration quite a bit. In fact, to migrate the 48K URLs into the site with a standard delay of 3 seconds would take about 40+ hours. That clearly wasn’t going to work.I actually went down this road as there is documentation from Jadu about how URLs are generated and I had success doing this for a different (non-Jadu) project.Here is the outline of the unit test URL mocking class.
Solving The URL Problem
Here is the class I generated, sans the 40K+ URLs of course.I’m quire sure that this makes creating anything useful in the API a real pain since referring back to the site needs to be done with manually placed links, but it’s clearly like this by design. I couldn’t find any documentation on why it is like this, but it almost feels like vendor lock-in. Please correct me if I’m wrong here. public function testJaduUrlFind(string $id, string $type, string $expectedResult, ?string $parentId = NULL): void {
$jaduUrl = MockJaduUrls::findUrlById($id, $type, $parentId);
$this->assertEquals($expectedResult, $jaduUrl);
}
The test function is pretty simple and consists of the following.<?php
declare(strict_types=1);
namespace Drupaljadu_migrateService;
/**
* Lookup class for the Jadu URLs.
*/
class JaduUrls extends BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [
// A whole load of URLs.
];
}
$url = JaduUrls::findUrlById("123", 'localgov_news_article');
Pulling data from the Jadu siteBefore I get into how I solved this issue let’s look at some of the methods I looked at to show what didn’t work.This is the second article in a series of articles looking at migrating from Jadu into a LocalGov Drupal (LGD) site for Central Bedfordshire. In the first article we looked at the Jadu API and setting things up so that we could make calls to the API and parse the XML data using the migration systems available.<?php
declare(strict_types=1);
namespace Drupaljadu_migrateService;
/**
* Lookup class for the JaduUrls.
*/
class BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [];
/**
* Find a URL by an ID and the type of page.
*
* @param string $id
* The ID.
* @param string $type
* The type of page.
* @param string|null $parentId
* An optional parent ID. This is used for guides pages especially.
*
* @return string|null
* The path, or null if nothing was found.
*/
public static function findUrlById(string $id, string $type, ?string $parentId = NULL): ?string {
// Set the parent ID place to be 0, which will never match.
$parentIdPlace = 0;
// Create the URL information from the paths.
// Note that the path contains a starting "/" so the counts here are all
// increased by 1 to allow for that.
switch ($type) {
case 'localgov_services_landing':
// Generated from category pages.
// URL format is:
// "/info/<id>/page_title".
$urlStart = 'info';
$urlParts = 4;
$pathIdPlace = 2;
break;
case 'localgov_news_article':
// Generated from news pages.
// URL format is:
// "/news/article/<id>/page_title".
$urlStart = 'news';
$urlParts = 5;
$pathIdPlace = 3;
break;
////////////////////////////////
// More types of page added here.
////////////////////////////////
}
if (!isset($urlParts) || !isset($urlStart) || !isset($pathIdPlace)) {
// Something we don't know about.
return NULL;
}
foreach (static::JADU_URLS as $url) {
$parts = explode('/', $url);
if ($parts[1] === $urlStart && count($parts) == $urlParts) {
if ($parentId !== NULL) {
if ($parts[$parentIdPlace] == $parentId && $parts[$pathIdPlace] === $id) {
return $url;
}
}
elseif ($parts[$pathIdPlace] === $id) {
if (isset($parts[3]) && $parts[3] === 'a_to_z') {
// This produces the wrong URL for the page.
continue;
}
return $url;
}
}
}
return NULL;
}
}
In addition to what you see here I also created a Drush command that generated a report on the pages in Drupal and all of the redirects created for those pages. This allowed us to see if any gaps existed in the redirects before we launched as we knew that every page created in Drupal after the migration must have at least one redirect one created, which is from the original Jadu URL.According to the documentation, Jadu URLs can be generated from the title of the page, using the following rules:Here is a section of that class, containing the logic for finding the original URL of a service landing page or a news page.<?php
declare(strict_types=1);
namespace Drupaljadu_migrateEventSubscriber;
use DrupalCoreDatabaseConnection;
use DrupalCoreEntityEntityTypeManagerInterface;
use Drupaljadu_migrateServiceAliasRedirectInterface;
use DrupalmigrateEventMigrateEvents;
use DrupalmigrateEventMigratePostRowSaveEvent;
use DrupalmigrateMigrateLookupInterface;
use DrupalmigratePluginMigrationPluginManagerInterface;
use DrupalnodeNodeInterface;
use SymfonyComponentEventDispatcherEventSubscriberInterface;
/**
* Migration event subscriber for Jadu migrate.
*/
class MigrationEventSubscriber implements EventSubscriberInterface {
/**
* Creates a MigrationEventSubscriber object.
*
* @param DrupalCoreEntityEntityTypeManagerInterface $entityTypeManager
* Drupal entity type manager.
* @param DrupalmigrateMigrateLookupInterface $migrateLookup
* Drupal migration lookup service.
* @param DrupalmigratePluginMigrationPluginManagerInterface $migrationPluginManager
* Drupal migration plugin manager.
* @param Drupaljadu_migrateServiceAliasRedirectInterface $aliasRedirect
* Drupal alias redirect service.
* @param DrupalCoreDatabaseConnection $database
* Drupal database connection.
* @param Drupaljadu_migrateServiceFixMarkup $fixMarkupService
* Fix markup service.
* @param Drupaljadu_migrateServiceWysiwygFileInterface $wysiwygFile
* Wysiwyg file service.
*/
public function __construct(
protected EntityTypeManagerInterface $entityTypeManager,
protected MigrateLookupInterface $migrateLookup,
protected MigrationPluginManagerInterface $migrationPluginManager,
protected AliasRedirectInterface $aliasRedirect,
protected Connection $database
) {}
/**
* Get subscribed events.
*
* @inheritdoc
*/
public static function getSubscribedEvents(): array {
$events[MigrateEvents::POST_ROW_SAVE][] = ['onMigratePostRowSave'];
return $events;
}
/**
* Triggered after a row is saved in the migration.
*
* @param DrupalmigrateEventMigratePostRowSaveEvent $event
* The event object.
*/
public function onMigratePostRowSave(MigratePostRowSaveEvent $event): void {
$migrationId = $event->getMigration()->getBaseId();
if ($migrationId === 'news') {
$row = $event->getRow();
$destination = $event->getDestinationIdValues();
$this->aliasRedirect->createRedirect(
'localgov_news_article',
(int) $row->getSourceProperty('id'),
(int) $destination[0]
);
}
}
}
<?php
declare(strict_types=1);
namespace DrupalTestsjadu_migrate;
use Drupaljadu_migrateServiceBaseJaduUrls;
/**
* Creates a test class with a known set of URLs to test with.
*/
class MockJaduUrls extends BaseJaduUrls {
/**
* The URLs.
*/
public const array JADU_URLS = [
'/news/article/123/test-test-test',
// More URls here to test.
];
}
In the last article I mentioned something about the Jadu API that caused me a lot of headaches. The API contains most of the information for a page, but critically, the Jadu API contains no information about the path of a page. There is basically no way to get the URL of a page in Jadu from the XML API.As I mentioned above, when generating new content on the site we used the Drupal and LGD path auto systems to generate the paths for the content. This meant that we didn’t have to change how the paths are generated in LGD, which would create a maintenance issue as the site would have deviated from the core path structure.Migrating redirects is never simple, but it is also never the end of the process. You should always look at any 404 messages coming from the site to ensure that you add redirects for any missing content to relevant pages.Migrating URLs into a site can be done in two ways. If you know the old and the new URLs you can just migrate directly into the redirect system since that’s a known quantity. Due to us not knowing this structure exactly before hand I therefore had to use the second method, which is to hook into the migration and run the needed logic to generate the redirects as the content is created. In order to do this we need to use the post row save event so that we can add redirects to the site after the new item of content has been added.Constructing the Jadu URL from information in the API XMLThe solution was to pull together a list of URLs and compile them into a data structure that I could then use during the migration.
Conclusion
I could have changed things slightly so that the URLs could have been read from a CSV file or a database table, but both of those mechanisms require extra work, extra code, and pre-migration setup to get working. In my experience, once the migration is done there is normally a bit of a gap where you concentrate on working on other bits of the site. I had lots of good documentation on how the migration was created, setting things up, and running the migration, but having extra steps in the migration would have complicated things further than they needed to be.Every time we created a page we needed to also create a redirect between the old Jadu URL and the new Drupal one. This allows the new site to function in the same way as the old site so that all search engine listings, external links, and any bookmarks still worked with the new content. A critical consideration, especially for a public facing government site as this is.If you are looking to migrate from Jadu, or are trying to get to grips with the migration system then please get in touch!Creating a massive array in a PHP class meant that it was as simple as it could be. I didn’t need to rely on making sure a database table was up to date or make sure that a CSV file was in the right place. I could also use the Drupal unit testing functions to test the code without mocking any Drupal services.Unfortunately, from testing the migrated content with known Jadu URLs I found that this method had about a 60% success rate, which isn’t good enough. The main issue is that these instrcutions are clearly incorrect, many URLs on the site do not remove uppercase characters, or strip out dashes or stop words. There were even a few cases where special characters were still present in the path.This mess of URLs was processed and collected together into a single array so that I could filter out any duplicates and make sure they were all of the same format. The array was then used to generate a PHP class that contained a single, static, property that contained all of the URLs available. I used a simple PHP script to generate this class, but the result was that I could add any number of URLs to the list and ensure that everything was accounted for and that I could easily add more URLs to the list.The AliasRedirect service class takes the DrupalredirectRedirectRepository class as a single dependency. It can then use that redirect service to find out if a redirect exists and to generate a new redirect for a given path. The path is found by using the JaduURls class created above. Much of the code in this class is used to ensure that the redirection path is good and that we aren’t creating duplicate redirects.Finally, there is also the issue of hierarchical pages. It’s not possible to find out the position of a page in the hierarchical structure of the site based on the metadata and URL of the page.In this article we will look at methods used to find the URLs of the Jadu site before moving onto the solution that solved the problem. We will then look at how those URLs are used in the migration of the site.In the next article I will look at migrating document and document content from Jadu into Drupal.
