Drupal 10: Testing Migration Process Plugins

For example, if you wanted to test something with a User entity then you would install the user module, which makes sense. But you would also need to install the user entity schema in order to generate the tables needed for the entity to exist.This is the full unit test class for the reformat_title migrate process plugin, which lives in the test/Unit/Plugin/migrate/process directory within our migration module.<?php

}


The reformat_title migrate process plugin doesn’t require any additional dependencies and as such can be tested using a unit test.The plugin.manager.migration service comes with a method called createStubMigration(), which we can use here to generate a simple test migration. This migration isn’t actually going to be used here (this is a process plugin) so it just needs the bare minimum setup.// Create the test row and executable objects.
$this->row = $this->getMockBuilder('DrupalmigrateRow')
->disableOriginalConstructor()
->getMock();
$this->migrateExecutable = $this->getMockBuilder('DrupalmigrateMigrateExecutable')
->disableOriginalConstructor()
->getMock();

reformat_title – The data we have from the source has the titles with markup in them, which we want to remove. Also, lots of pages in the source data have “page” in the title so we want to remove this word wherever it occurs. As a final step, the title should have the first letter of each word capitalized.Once you get used to setting up unit and kernel testing classes as required for your migrate plugins you can look at using test driven development to create them. Using test driven development helps to write the code you need without actually running a migration.I have been involved in migrations on sites with LOTS of data; often involving lots of dependencies to look up field values. This can mean that in order to test the migration you are looking at hours of setup to get to the point where you can see your migration in action. By using unit tests for your process plugins you can quickly update your data provider with edge cases, work to solve them in your plugin, and commit your changes.Data providers are PHPUnit plugins that will call our test case multiple times, each time with a different test setup. To set them up you just need to add the @dataProvider property to your test docblock comment, and define the method that will be used for the data provider.

• Testing your migrate process plugins is a very powerful tool when writing complex migration. Not only can it help you spot and correct edge cases in the source data, but it also can help speed up your migration development.<?php

  namespace DrupalTestsmigration_process_testUnitPluginmigrateprocess;

  use Drupalmigration_process_testPluginmigrateprocessReformatTitle;
  use DrupalTestsmigrateUnitprocessMigrateProcessTestCase;

  /**
  * Tests the reformat_title migration plugin.
  */
  class ReformatTitleTest extends MigrateProcessTestCase {

  /**
  * Test that different title values reformat correctly.
  *
  * @dataProvider titleIsReformattedDataProvider
  */
  public function testTitleIsReformatted($sourceValue, $expectedResult) {
  $plugin = new ReformatTitle([], 'reformat_title', []);
  $value = $plugin->transform($sourceValue, $this->migrateExecutable, $this->row, 'title');
  $this->assertEquals($expectedResult, $value);
  }

  /**
  * Data provider for testTitleIsReformatted.
  *
  * @return array
  * The data to be tested.
  */
  public function titleIsReformattedDataProvider() {
  return [
  [
  '<p>About us page</p>',
  'About Us',
  ],
  [
  '<p>contact Us page</p>',
  'Contact Us',
  ],
  ];
  }

  }


  <?php

  namespace Drupalmigration_process_testPluginmigrateprocess;

  use DrupalCoreConfigImmutableConfig;
  use DrupalCorePluginContainerFactoryPluginInterface;
  use DrupalmigrateMigrateExecutableInterface;
  use DrupalmigrateProcessPluginBase;
  use DrupalmigrateRow;
  use SymfonyComponentDependencyInjectionContainerInterface;

  /**
  * Fix any broken markup in the source field.
  *
  * @code
  * body/0/value:
  * plugin: fix_data_content
  * @endcode
  *
  * @MigrateProcessPlugin(id = "fix_data_content")
  */
  class FixDataContent extends ProcessPluginBase implements ContainerFactoryPluginInterface {

  /**
  * The config factory object.
  *
  * @var DrupalCoreConfigImmutableConfig
  */
  protected ImmutableConfig $siteConfig;

  /**
  * {@inheritDoc}
  */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
  $instance = new self($configuration, $plugin_id, $plugin_definition);

  $instance->siteConfig = $container->get('config.factory')->get('system.site');

  return $instance;
  }

  /**
  * {@inheritDoc}
  */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
  if ($value === NULL) {
  return $value;
  }

  // Strip any empty elements.
  $value = preg_replace('/<span>s*</span>/', '', $value);
  $value = preg_replace('/<p>s*</p>/', '', $value);

  // Replace all instance of [email protected] with our site email.
  $value = preg_replace('/[email protected]/', $this->siteConfig->get('mail'), $value);

  return $value;
  }

  }


  In the case of this plugin we want to get access to a set of configuration, so we grab the config.factory service and extract the system.site configuration, which we store in a variable.

This migrate plugin has no dependencies and so will be tested using a unit test setup.

If you want the full source code of this module then it available on GitHub. Feel free to use it as a skeleton when setting up your custom migrate plugins.

Creating The Reformat Title Migrate Plugin

Both the core Migrate module and the excellent Migrate Plus module contain a number of different process plugins that you can use to process your data in different ways.The plugin.manager.migrate.process service is aware that we might want to have dependencies injected into the object and as such will look to see if the plugin class extends the interface DrupalCorePluginContainerFactoryPluginInterface. If it does then the plugin manager will call a static create() method within the class that will create the object and inject any dependencies.The fix_data_content migrate process plugin has the same setup as the reformat_title plugin, with one exception. We want to inject a dependency into this plugin and so we can use a Drupal service.To create the process plugin as an usable object we just need to call it, the constructor (defined in DrupalComponentPluginPluginBase) requires some simple arguments to instantiate the object, but as none of them are objects we don’t need to do any mocking.$this->migrateLookup->method('lookup')->willReturn([['mid' => 1]]);

Process plugins are responsible for copying and sometimes manipulating data into the destination. There are a number of different process plugins that allow you to get data in different ways and and apply it to your destination fields.

Unit Testing The Reformat Title Migrate Plugin

The test/Kernel/Plugin/migrate/process directory within our custom migration module.$value = $plugin->transform('<p>About us page</p>', $this->migrateExecutable, $this->row, 'title');
$this->assertEquals('About Us', $value);

This migrate plugin requires the site configuration factory (config.factory) to be injected as a service so that we can get hold of the site email address. As this is a more involved plugin setup we will use a kernel test to test this plugin$value = $plugin->transform('<p>Some text.<span></span></p>', $this->migrateExecutable, $this->row, 'field_body');
$this->assertEquals('<p>Some text.</p>', $value);

With regards to our fix_data_content migrate process plugin, because it uses Drupal configuration there it a little bit more setup to manage before we can get to actually running any tests. As Drupal is minimally bootstrapped we do have access to some of the core services within the system, one of which being the configuration system.In this article we will look at two custom migrate process plugins that are built in different ways and how to test them. This will dive into some concepts around Drupal plugin management, dependency injection, as well as unit testing and data providers with PHPUnit.We aren’t using any configuration parameters for these process plugins in order to keep things as simple as possible.$migratePluginManager = $this->getMockBuilder('DrupalmigratePluginMigrationPluginManager')
->disableOriginalConstructor()
->getMock();
$this->migrateLookup = $this->getMockBuilder('DrupalmigrateMigrateLookup')
->setConstructorArgs([$migratePluginManager])
->getMock();

A kernel test differs a little from a unit test in that it allows you to minimally bootstrap Drupal. During the setup of the test a minimally installed Drupal site will be setup and you can then augment this with the modules, entities, and schemas that you need. This includes installing the module you are currently testing.fix_data_content – Like many migrate sources, the source data for body fields is pretty messy. There are tags containing no text that we want to remove from the markup before it is added to the site. A number of hard coded contact email addresses have also been added to the markup, which we want to swap for the email address for our new site

Creating The Fix Data Content Migrate Plugin

First, let’s look at the migration script that we will be using in this article. All of the source code for this migration example is available on GitHub.The entire plugin class for the reformat_title plugin isn’t that big, it just contains the annotation and the transform method.In the migrate script we defined the reformat title plugin to have the id “reformat_title”, so we need to create a class that extends DrupalmigrateProcessPluginBase in the location src/Plugin/migrate/process.@MigrateProcessPlugin(id = "reformat_title")

Another consideration that we haven’t touched on above is the ability to mock services in your kernel tests. This is useful when testing plugins as you can mock any calls to the database in order to return known data to your unit tests.Creating a process plugin in migrate is quite easy (thankfully). Drupal plugin management system is used to pick up the plugin name using an annotation and generate the plugin object we can use.

Kernel Testing The Fix Data Content Migrate Plugin

A common example of this is mocking the migrate.lookup service, which is used to lookup mapped values from the source data to the destination data. To mock this class we would do something like the following.$container = Drupal::getContainer();
$container->set('migrate.lookup', $this->migrateLookup);
Drupal::setContainer($container);

If your process plugin needs to pull some values out of the row object then you can use the global property to mock methods and return values as you normally would.To generate the migrate process plugin in our test case we need to use the plugin.manager.migrate.process service. The createInstance() method of this plugin will automatically call the create() method in our plugin class, which we used to generate the dependencies we require for the plugin to work. In order to call the createInstance() method we need to supply a couple of extra properties. These are a migration object and any configuration we want to pass to the plugin itself.As a side note, I actually struggled to find an example that would be simple enough to demonstrate the concepts involved, but not too complex that it required a bunch of other services and dependencies to get working. To this end I decided that a simple email address swap would be the simplest thing we could do. We just need to grab the site email from the Drupal configuration, which means injecting the config.factory service into the plugin.First, we need to setup the configuration so that the system.site.mail configuration setting contains a known value. To do this we just ask Drupal for the config.factory service and change the value we are going to use in the test.id: migration_process_test
label: Testing custom migration process plugins

destination:
plugin: entity:node
default_bundle: page


The destination of this migration is the Page content type, which we get when using the Standard Drupal install profile.In order to unit test a migrate process plugin we need to extend the class DrupalTestsmigrateUnitprocessMigrateProcessTestCase. This class extends the Drupal core DrupalTestsUnitTestCase class and adds some boiler plate code so that we can instantiate the plugin object without having to write our own mocking code.destination_field: source_field

If you have any comments or questions about migration plugins then please let me know in the comments below. Or, get in touch via the contact form for a more involved look.// Update the site configuration with our test email address.
Drupal::service('config.factory')->getEditable('system.site');
$system = $this->config('system.site');
$system
->set('mail', '[email protected]')
->save();

As I mentioned earlier, the transform() method of the process plugin, which is what the migrate system will execute during the migration, requires two objects from the migrate system. These are the current migrate system executable and an object representing the current row being processed. The MigrateProcessTestCase object will create two properties that we can use to call transform() and not worry about where to get those objects from.We will use the embedded_data migrate source plugin so that we can embed the source data directly in the migrate script itself. This plugin is useful for very quick migrations, but also benefits us here as we can use it as a pedagogical device. The source data here is intentionally messy, to simulate a messy migration source.

Mocking Services In Kernel Tests

Drupal’s migration system allows the use of a number of different plugins to perform source, processing, and destination functions. protected static $modules = [
'migrate',
'migration_process_test',
'system',
];

We add the following annotation to this class to announce that it is a process plugin. All process plugin classes must contain this.Out of the box, the default process plugin is the get plugin, which can be used like this in your migration scripts.protected function setUp(): void {
parent::setUp();
}

The full class for the fix_data_content migrate process plugin isn’t that large. In fact, it mostly consists of the boilerplate code needed to setup the object with the dependencies we require.Kernel tests in Drupal extend the DrupalKernelTestsKernelTestBase class. When run, this class will automatically detect a class property called $modules, which it will use to determine the list of modules that must be installed. In our case, we just need the core migrate module, our own custom module (called migration_process_test) and the system module for the core system and configuration setup.// Create migration stub.
$migration = Drupal::service('plugin.manager.migration')
->createStubMigration([
'id' => 'test',
'source' => [],
'process' => [],
'destination' => [
'plugin' => 'entity:node',
],
]);

// Generate the plugin via the plugin.manager.migrate.process service.
$plugin = Drupal::service('plugin.manager.migrate.process')
->createInstance('fix_data_content', $configuration, $migration);

destination_field:
plugin: get
source: source_field

$plugin = new ReformatTitle([], 'reformat_title', []);

With the plugin object in hand we can then call the transform() method, passing in the source value we want to test and the migrate executable and row arguments generated in the parent class. This means that our test can be boiled down to the following two lines.