UE5 Data Driven Tool Development

In recent works, I find that tool development needs to be data oriented. This is not only useful for quick query, keeping the current work intact and can be resumed, but also be shared with others.

In recent works, I find that tool development needs to be data oriented. This is not only useful for quick query, keeping the current work intact and can be resumed after, but also the data can be shared with the people within the project to be revised and decide on what the next courses of actions are.

Below is a prime example.

Deprecated, expired or archive asset is a common term. These are the assets that are no longer being used in the project, but still exists in the editor. There can be various reasons, and mostly was because of the creative process and coping with deadlines so people moving on, without bothering about these remnants of creativeness.

Thus, results in large amounts of unuse assets. And in the case of over a decade of development, these items can be thousands.

Not only did they create confusion for the new comer, they take up a large space in the project, and accumulate on the loading time whenever someone tries to run the editor or level.

So, we need to find and eliminate them (Of course, we only need to delete those that aren't actually used).

But we cannot go over thousands of Reference Viewer and check.

Ideally, there should be a list that displays all these relationships and picks out those without any reference.

The list could be like below, I thought:

Similar to the Reference Viewer, each row displays an asset in an object form,

And it contains a number of asset objects, in the project, in a subfolder, or based of selected assets in the Content Browser.

From this, we could create a mock up data struct:

Something like this:

And from this struct, we can create a new data table and make our first row of data using the reference viewer example above:

Now the final step is to figure out a way to record Unreal data and put into this table.

The problem is, at the moment, I could not find a way to directly write the data into a data table asset. As far as I know, DT pure purpose is to be read not be written. So, out of the box, this function does not exists.

However, we could make a data table by import a json or csv file.

Now we have a recipe and ingredients that we need. Let's cook.

First, we gather the input and iterate through:

Set the object to be the target asset, our first column in the table:

Next, we gather the references of this "target object" and find the amount of references it is being used (second and third columns)

Gotcha: It is worth noting that to get the reference, you must provide the Asset Registry, package path to the target object and optionally the reference options.

What I find useful is to to create asset data from an object. So that we can have the "identify" information (path, name, object path) that functions require:

Likewise for the target object's dependencies (5th and last column):

As you might have notice, the data is recorded into a struct. And this struct is the one that I introduce in the table above. Same representation, same concept of the table. These structs now represent row in big table. And need to be append to.

In this case, I use an array. After every target object get iterated, the object gets to be added to the array:

The full graph would be like below:

You may notice that a few arrays must be clear before their items get added. By default, Unreal Blueprint data doesn't get clear up after every run. Without this step, the data keeps on added and would slow down the operation. So, it is good to clear them before using.

This operation gets call from a button as shown here:

The user would need to select certain assets in the browser, and click this button to trigger:

What I often use for ease of debugging is using the Details View to display the data that I want (rather than using a debug log):

The data is given as below:

Then, we can use this array to iterate over and write them as json (using Python scripting):

Though, we need to figure out what this json schema would look like, by exporting the datatable in the json format:

I then start to create an operation that results in something like the above:

In every parentheses, there is a variable serves as an input:

However, we need actual parentheses that convers a json object like the format shown above, so we can replace these strings with symbols:

And eventually, we can pipe the output string to be written as json file using Python:

I split out the file name to be an input (something we can provide for the user to change)

And like above, we can add this operation to be a button that the user can run:

Gotcha: this is an optional step, we can add a fill json operation to update the data table (you can do this manually by import the json file into the data table as wish):

In this case, I create a new data table and import the json file in. The result from CreateAsset is a wild card, which is in compatible with Data Table input from the Fill Data node, so it must be "Cast to Data Table".

Finally, you can iterate through this data table, picking out those without and references and delete the asset:

As a result, all the target objects are deleted, yet their column in the data table exist:

This is extremely useful for when you want to revise upon, and with a Source Control, you could bring them back after a "Remove assets" CL is submitted.

Eventually, you could share this table with anyone in the project team, having them revise and delete the remainings items. Those with more than one references, can be remove from a level or a BP before deleting. Or you could iterate through, and assign a visually unique material for identifying in the level.

The idea goes on and on.

And that's is one example of using data to drives tool development in UE5.