Serenity 6.5.2 Release Notes (2023-02-21)

Revamped Reporting and Html to Pdf System [Breaking Change]

Serenity has a flexible reporting system. Out of the box, we have HTML-based reporting which is converted to PDF via WKHTMLTOPDF, Data only reports that are exported to XLSX files via EPPlus, external reports that launch a custom URL, etc.

All these report types were abstracted so you did not have to directly interfere with the underlying tools used in your reports, and they could be replaced with alternatives in the future if required.

Since we moved the classes like the ReportController to the Serenity.Extensions package to make them easier to update, it became a bit harder to intercept the process, define custom report types, or replace the underlying tools.

For example, there was a switch statement in ReportController that looks like this:

if (report is IDataOnlyReport dataOnlyReport)
    // render data only report via EPPlus
else if (report is IExternalReport externalReport)
    // render external report as a URL redirect
else 
    // render html report via WKHtmlToPdf

As these statements are in an assembly from a NuGet package, it is not possible to inject custom report handling here.

We now abstracted this process; and many other sub-processes into interfaces like the following:

• IReportRenderer: Renders a report object to the target format, e.g. the above switch statement
• IReportRetrieveHandler: Retrieves details about a report type, e.g. its parameters, etc.
• IReportFactory: Creates an instance of a report type, optionally setting its parameter values
• IHtmlToPdfConverter: Executes WKHTMLToPdf, Puppeteer, etc. for HTML to PDF conversion
• IHtmlToPdfRenderer: Renders an HTML report to PDF, via IHtmlToPdfConverter service.
• IHtmlReportRenderUrlBuilder: Generates a rendering callback URL for an HTML report. To convert an HTML report to PDF, the external reporting tool (WKHTML, Puppeteer, etc.) needs to call back into a URL in the site. It's this abstraction's responsibility to generate that URL.

The default implementations for these interfaces need to be registered in the service provider, so you'll need to replace the following line in Startup.cs:

services.AddExcelExporter();

with:

services.AddReporting();

Introduced Puppeteer (Chrome/Firefox) Based Html to Pdf Converter [StartSharp]

Serenity uses WKHTMLToPdf by default to convert HTML reports to PDF. WKHtmlToPdf is not updated for some time, and it does not support modern CSS features like Flexbox, Grid layout, etc. So it is not compatible with Bootstrap 5's layout system. To workaround that you could use Bootstrap 3 on your report pages.

Also, as it is an executable, it needs to be manually installed in the target server, alongside its dependencies like Visual C++ runtime.

There is now a Puppeteer option in Serenity.Pro.Extensions, which use Chrome (or Firefox). To enable that, just add the following line before AddReporting:

services.AddPuppeteerHtmlToPdf();
services.AddReporting();

Chrome (or Firefox) will be automatically downloaded to App_Data/chrome or App_Data/firefox directory on the first report execution. If you would like to customize the folder to download the browser, you can do so via the appsettings.json file:

"PuppeteerHtmlToPdf": {
    "DownloadPath": "C:/somedir"
}

Also, add the following in Startup.cs:

services.Configure<PuppeteerHtmlToPdfSettings>(Configuration.GetSection(PuppeteerHtmlToPdfSettings.SectionKey));

If there are some reports that you would like to still use WKHtmlToPdf, add the following attribute:

[UseWkHtmlToPdf(true)]
public class MyLegacyReport
{
}

New Expression Attributes

In Serenity, entities can work like SQL views, without having to define an underlying SQL view.

Let's say we have a Person entity with Firstname and LastName properties:

public class PersonRow : Row<PersonRow.RowFields>
{
    public int PersonId { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

And we wanted to define a calculated FullName property:

public class PersonRow : Row<PersonRow.RowFields>
{
    public int PersonId { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    [Expression("(T0.FirstName + ' ' + T0.LastName)")]
    public string FullName { get; set; }
}

It is possible to define an expression field (or view/calculated field) via the Expression attribute.

T0 here is a default alias used by Serenity rows, which corresponds to the main table, e.g. the Person table.

As Serenity, and its demo has to support multiple database types, the expression which is written using the plus (+) operator has a problem. It is only supported for the Microsoft SQL Server.

Luckily, there is a CONCAT function, which we could use instead:

[Expression("CONCAT(CONCAT(T0.[FirstName], ' '), T0.[LastName])")]
public string FullName { get; set; }

Even though CONCAT is supported by many database types, at least for the version that accepts two arguments, Sqlite does not have that.

To solve that, we can use Serenity's ability to select expressions based on the target dialect, e.g. the connection's dialect:

[Expression("CONCAT(CONCAT(T0.[FirstName], ' '), T0.[LastName])")]
[Expression("(T0.FirstName || ' ' || T0.LastName)", Dialect = "Sqlite")]
public string FullName { get; set; }

Serenity will automatically select the second expression with pipe (||) operators for Sqlite, while the first one with the CONCAT function for others.

Anyway, whenever we need such a string concatenation operation, we'll have to repeat these expressions over and over. It's easy to make typos or forget which database supports which expression.

And the worst part is even the CONCAT function has differences among server types. While CONCAT('A', null) returns A for some server types, others might result in null.

So, for example, if a person has FirstName, MiddleName, and LastName properties and you use CONCAT for them to build the full name, as most people don't have a middle name, you'll end up with a null FullName for some servers.

In this version, we introduced some custom expression types, and the ability for expression attributes to produce different expressions based on the target dialect.

One of them is the [Concat] attribute:

[Concat("T0.[FirstName]", "' '", "T0.[LastName]")]
public string FullName { get; set; }

This attribute will build its final expression based on the target dialect. So for Sqlite it will use the pipe operator, and for others, it will use the CONCAT function.

As a bonus, it will also wrap the operands with COALESCE(operand, '') statements so that you will no longer have to worry about NULLs. You can turn off that behavior by setting NullAsEmpty to false:

[Concat("T0.[FirstName]", "' '", "T0.[LastName]", NullAsEmpty = false)]
public string FullName { get; set; }

Currently, we have the following expression attributes:

• Case: The good old CASE WHEN a THEN 'x' WHEN b THEN 'y' ELSE 'z' END
• CaseSwitch: CASE statement with a switch value, e.g. CASE SomeValue WHEN 1 THEN 'x' WHEN 2 THEN 'y' ELSE 'z' END
• Concat: String concatenation using CONCAT where possible
• DateDiff: e.g. SQL Server datediff function
• DatePart: e.g. SQL Server datepart function
• SqlNow: e.g. SQL Server sysdatetime() function
• SqlUtcNow: e.g. SQL Server sysutcdatetime() function
• SqlDateTimeOffset: e.g. SQL Server sysdatetimeoffset() function

All the attributes, including the Expression attribute, has now a base attribute type:

namespace Serenity.Data.Mapping;

[AttributeUsage(AttributeTargets.Property, AllowMultiple = true)]
public abstract class BaseExpressionAttribute : Attribute
{
    public abstract string Translate(ISqlDialect dialect);
    public string ToString(ISqlDialect dialect);
    public string Format { get; set; }

    public static string ToString(object expression, ISqlDialect dialect);
}

If you wanted to create another custom expression attribute, you need to override the Translate function.

The ToString(dialect) method calls the Translate method while handling some extra implementation details like the ability to apply an optional format string (the Format property) to the expression produced by the Translate method and giving the ability to the dialect to override the conversion process by implementing the ISqlExpressionTranslator interface:

public string ToString(ISqlDialect dialect)
{
    string expression;
    if (dialect is ISqlExpressionTranslator translator)
    {
        expression = translator.Translate(this) ??
            Translate(dialect);
    }
    else
        expression = Translate(dialect);

    if (string.IsNullOrEmpty(Format))
        return expression;

    return string.Format(Format, expression);
}

Currently, none of the out-of-the-box dialects implement it, but we added that ability anyway, in case some users might want to customize the translation process for their custom dialect types.

Improvements and Fixes to Grouping / Summaries

SlickGrid (and our rewrite SleekGrid) is very flexible and extensible. It can work with any data source be it in-memory, remote, or any other type. It can be extended via a simple plugin interface.

For example, SlickGrid itself (slick.core.js + slick.grid.js) knows nothing about the grouping, group rows, or totals. It is completely handled by the data source (e.g. view).

The data view in SlickGrid (called RemoteView in Serenity) has to handle the grouping and calculation of the summaries.

After this, a plugin is required to make the group rows look like they should (e.g. with group title, value, and item count), and display the footers with calculated summaries. That plugin is called GroupItemMetadataProvider (slick.groupitemmetadataprovider.js).

Unlike an ordinary SlickGrid plugin, the GroupItemMetadataProvider has to be registered in both the grid and the view as it has to coordinate some metadata and operations between them.

Please note that, even though in Serenity grids (DataGrid), there is a one-to-one relation between a data source and its grid, it is normally possible to use a data source by multiple SlickGrid instances.

In our basic grouping sample (in the common-features repo) we had something like this:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    protected createSlickGrid() {
        var grid = super.createSlickGrid();

        // need to register this plugin for grouping or you'll have errors
        grid.registerPlugin(new GroupItemMetadataProvider());

        //...
    }
}

If you don't create a GroupItemMetadataProvider and register it as a plugin in the SlickGrid instance, the group rows won't display and you may get errors.

Unfortunately, there is a problem with the sample above. Because the plugin is not registered in the data view (RemoteView), it has no idea there is already a GroupItemMetadaProvider, so it creates a new one when grouping is initialized:

function setGrouping(groupingInfo) {
    if (!options.groupItemMetadataProvider) {
        options.groupItemMetadataProvider = new Slick.Data.GroupItemMetadataProvider();
    }
    //...

The code block above is from the slick.dataview.js file in the original SlickGrid repository. We replicated a similar logic in our RemoteView class.

So, if you don't pass the plugin to view options, there are two instances of the same plugin lying around. One is registered with the grid, while one is not as it is internally created by the data view.

This was not a big problem as long as the GroupItemMetadataProvider is created with no options. Otherwise, the instance we created and the one data view internally created would have a different set of options.

Because of this, even though we forgot to pass the GroupItemMetadataProvider to the data view, our sample used to work. But it stopped working after we made some refinements to the GroupItemMetadataProvider class, resulting in this issue:

• https://github.com/serenity-is/Serenity/issues/6692

The correct code should have been written like this:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    private groupItemMetadataProvider: GroupItemMetadataProvider;

    protected getViewOptions() {
        var opt = super.getViewOptions();
        this.groupItemMetadataProvider = new GroupItemMetadataProvider({ 
            // any extra options you want to pass
        });
        opt.groupItemMetadataProvider = this.groupItemMetadataProvider;
        return opt;
    }

    protected createSlickGrid() {
        var grid = super.createSlickGrid();
        grid.registerPlugin(this.groupItemMetadataProvider);
        //...
    }
}

This would register the same instance in both the grid; and the view.

With the latest SleekGrid (1.5.5+), it is possible to get the passed GroupItemMetadataProvider instance from the view after it is created, so it is no longer necessary to have an extra private variable:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    protected getViewOptions() {
        var opt = super.getViewOptions();
        this.groupItemMetadataProvider = new GroupItemMetadataProvider({ 
            // any extra options you want to pass
        });
        opt.groupItemMetadataProvider = this.groupItemMetadataProvider;
        return opt;
    }

    protected createSlickGrid() {
        var grid = super.createSlickGrid();
        grid.registerPlugin(this.view.getGroupItemMetadataProvider());
        //...
    }
}

Our DraggableGroupingMixin in StartSharp is modified to intelligently create a GroupItemMetadataProvider if it is not already registered with the grid and also, register it with the view so it is now a one-liner:

export class DragDropGroupingGrid extends OrderGrid {

    protected createToolbarExtensions() {
        super.createToolbarExtensions();

        new DraggableGroupingMixin({ grid: this });
    }
}

This automatically creates a GroupItemMetadataProvider, registers it as a SlickGrid plugin, and sets the groupItemMetaProvider of the data view.

HtmlEncode Now Works Like AttrEncode Which is Deprecated

We had a Q.htmlEncode and a similar function for attribute values called Q.attrEncode.

attrEncode escaped also the quote characters (" and ') while htmlEncode did not as they are not normally required to be escaped for HTML markup other than the attribute values. Escaping the angle brackets and ampersands would be enough.

It seemed like some users mixed up two, and as it is not a noticeable performance effect to always escape the quotes, we decided to make them work the same.

New Q.localText Method Instead of Q.text

The function name text was confusing for ES module style TypeScript, as it is common to have a text variable, so we renamed it to localText. The old one is still available but is obsolete. Using localText() will also make it easier to find the code block where you used local texts.

[WARNING] Possible Script Injection via the Translation Screen and Local Texts

If htmlEncode is not called when using local texts in a script method (e.g. <div>{ text("SomeKey") }</div> instead of <div>{htmlEncode(text("SomeKey")) }</div>, like getTemplate, etc. an attacker may use the translations screen to inject javascript.

This affected parts of our demo as everyone could log in as the admin and can use the translation screen.

The translations are reset every 30 mins in our demo so it was not a big issue. But you are strongly recommended to check your code in addition to standard screens like the login, signup and forgot password, etc.

See our latest commits in Serene/StartSharp/Common Features, etc. repositories for the fixes we applied.

Even if no one other than the admin can enter the translations screen in your application, it is still a good idea to mitigate the risk as you should not trust translators, even the admin himself.

We can't change the "text", "localText" or "tryGetText" functions to htmlEncode by default, as they may be used by others in contexts other than HTML, like the functions that expect raw text, e.g. notify messages, dialog titles, column titles, etc., which would result in double HTML encoding.

Prefer the localText() function instead of the obsolete text() function for better discoverability (but remember that it does not escape as well)