Previous Page Next Page

Serenity 6.5.2 Release Notes (2023-02-21)

Revamped Reporting and HTML to PDF System [Breaking Change]

Serenity boasts a flexible reporting system. Out of the box, we offer HTML-based reporting that can be converted to PDF using WKHTMLTOPDF. We also provide data-only reports that can be exported to XLSX files through EPPlus, and external reports that can launch custom URLs, among other functionalities.

All these report types have been abstracted to ensure that you don't need to directly interact with the underlying tools used in your reports. This abstraction allows for easy replacement with alternatives in the future if necessary.

However, with the relocation of classes such as the ReportController to the Serenity.Extensions package for easier updates, it became slightly more challenging to intercept the process, define custom report types, or replace the underlying tools.

For example, there used to be a switch statement in the ReportController that looked like this:

if (report is IDataOnlyReport dataOnlyReport)
    // render data-only report via EPPlus
else if (report is IExternalReport externalReport)
    // render an external report as a URL redirect
else 
    // render an HTML report via WKHtmlToPdf

Since these statements are in an assembly from a NuGet package, it was not possible to inject custom report handling here.

To address this, we have abstracted this process, as well as many other sub-processes, into interfaces such as the following:

  • IReportRenderer: Renders a report object to the target format, similar to the above switch statement.
  • IReportRetrieveHandler: Retrieves details about a report type, including its parameters.
  • IReportFactory: Creates an instance of a report type, optionally setting its parameter values.
  • IHtmlToPdfConverter: Executes tools like WKHTMLToPdf or Puppeteer for HTML to PDF conversion.
  • IHtmlToPdfRenderer: Renders an HTML report to PDF using the IHtmlToPdfConverter service.
  • IHtmlReportRenderUrlBuilder: Generates a rendering callback URL for an HTML report. To convert an HTML report to PDF, the external reporting tool (WKHTML, Puppeteer, etc.) needs to call back to a URL in the site, and this abstraction is responsible for generating that URL.

To adapt to these changes, make sure to replace the following line in your Startup.cs:

services.AddExcelExporter();

with:

services.AddReporting();

Introduced Puppeteer (Chrome/Firefox) Based HTML to PDF Converter [StartSharp]

Serenity now offers a Puppeteer-based HTML to PDF converter in Serenity.Pro.Extensions. This new option is particularly useful for StartSharp users.

By default, Serenity utilizes WKHTMLToPdf for converting HTML reports to PDF. However, WKHTMLToPdf has not been updated for some time and lacks support for modern CSS features like Flexbox, Grid layout, and may not be fully compatible with Bootstrap 5's layout system. As a workaround, you could use Bootstrap 3 on your report pages.

Additionally, because WKHTMLToPdf is an executable, it requires manual installation on the target server, along with its dependencies like Visual C++ runtime.

To enable the Puppeteer option, add the following line before calling AddReporting in your code:

services.AddPuppeteerHtmlToPdf();
services.AddReporting();

Upon the first report execution, Chrome (or Firefox) will be automatically downloaded to the App_Data/chrome or App_Data/firefox directory. If you wish to specify a custom folder for downloading the browser, you can do so via the appsettings.json file:

"PuppeteerHtmlToPdf": {
    "DownloadPath": "C:/somedir"
}

Don't forget to include the following configuration in your Startup.cs:

services.Configure<PuppeteerHtmlToPdfSettings>(Configuration.GetSection(PuppeteerHtmlToPdfSettings.SectionKey));

If there are specific reports that you still prefer to use WKHtmlToPdf for, you can do so by adding the [UseWkHtmlToPdf(true)] attribute:

[UseWkHtmlToPdf(true)]
public class MyLegacyReport
{
}

New Expression Attributes

In Serenity, entities can function like SQL views without the need to define an underlying SQL view.

Let's consider a Person entity with FirstName and LastName properties:

public class PersonRow : Row<PersonRow.RowFields>
{
    public int PersonId { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
}

Suppose we want to define a calculated FullName property:

public class PersonRow : Row<PersonRow.RowFields>
{
    public int PersonId { get; set; }
    public string FirstName { get to; }
    public string LastName { get; set; }

    [Expression("(T0.FirstName + ' ' + T0.LastName)")]
    public string FullName { get; set; }
}

You can define an expression field (or view/calculated field) using the Expression attribute.

Here, T0 is the default alias used by Serenity rows, corresponding to the main table, i.e., the Person table.

However, because Serenity and its demo support multiple database types, the expression written using the plus (+) operator may encounter issues. It's only supported in Microsoft SQL Server.

Fortunately, there's a CONCAT function that we can use instead:

[Expression("CONCAT(CONCAT(T0.[FirstName], ' '), T0.[LastName])")]
public string FullName { get; set; }

Although CONCAT is supported by many database types, there are exceptions. For example, Sqlite does not support it.

To address this, we can utilize Serenity's capability to select expressions based on the target dialect, such as the connection's dialect:

[Expression("CONCAT(CONCAT(T0.[FirstName], ' '), T0.[LastName])")]
[Expression("(T0.FirstName || ' ' || T0.LastName)", Dialect = "Sqlite")]
public string FullName { get; set; }

Custom Expression Types and Target Dialect

Serenity will automatically select the second expression with pipe (||) operators for Sqlite, while the first one with the CONCAT function for other databases.

However, whenever we need such a string concatenation operation, we'll have to repeat these expressions repeatedly. It's easy to make typos or forget which database supports which expression.

The worst part is even the CONCAT function has differences among server types. While CONCAT('A', null) returns A for some server types, others might result in null.

For example, if a person has FirstName, MiddleName, and LastName properties, and you use CONCAT for them to build the full name, most people don't have a middle name, resulting in a null FullName for some servers.

In this version, we've introduced some custom expression types and the ability for expression attributes to produce different expressions based on the target dialect.

One of them is the [Concat] attribute:

[Concat("T0.[FirstName]", "' '", "T0.[LastName]")]
public string FullName { get; set; }

This attribute builds its final expression based on the target dialect. For Sqlite, it uses the pipe operator, and for others, it employs the CONCAT function.

As a bonus, it wraps the operands with COALESCE(operand, '') statements so that you will no longer have to worry about NULLs. You can turn off that behavior by setting NullAsEmpty to false:

[Concat("T0.[FirstName]", "' '", "T0.[LastName]", NullAsEmpty = false)]
public string FullName { get; set; }

Currently, we have the following expression attributes:

  • Case: The classic CASE WHEN a THEN 'x' WHEN b THEN 'y' ELSE 'z' END
  • CaseSwitch: CASE statement with a switch value, e.g., CASE SomeValue WHEN 1 THEN 'x' WHEN 2 THEN 'y' ELSE 'z' END
  • Concat: String concatenation using CONCAT where possible
  • DateDiff: e.g., SQL Server datediff function
  • DatePart: e.g., SQL Server datepart function
  • SqlNow: e.g., SQL Server sysdatetime() function
  • SqlUtcNow: e.g., SQL Server sysutcdatetime() function
  • SqlDateTimeOffset: e.g., SQL Server sysdatetimeoffset() function

All the attributes, including the Expression attribute, now have a base attribute type:


namespace Serenity.Data.Mapping;

[AttributeUsage(AttributeTargets.Property, AllowMultiple = true)]
public abstract class BaseExpressionAttribute : Attribute
{
    public abstract string Translate(ISqlDialect dialect);
    public string ToString(ISqlDialect dialect);
    public string Format { get; set; }

    public static string ToString(object expression, ISqlDialect dialect);
}

These enhancements provide better control and compatibility when working with different SQL dialects in Serenity.

Custom Expression Attributes and Overriding Translate Function

If you want to create another custom expression attribute, you need to override the Translate function.

The ToString(dialect) method invokes the Translate method while handling additional implementation details. It includes the ability to apply an optional format string (the Format property) to the expression produced by the Translate method and allows the dialect to override the conversion process by implementing the ISqlExpressionTranslator interface:

public string ToString(ISqlDialect dialect)
{
    string expression;
    if (dialect is ISqlExpressionTranslator translator)
    {
        expression = translator.Translate(this) ??
            Translate(dialect);
    }
    else
        expression = Translate(dialect);

    if (string.IsNullOrEmpty(Format))
        return expression;

    return string.Format(Format, expression);
}

Currently, none of the out-of-the-box dialects implement it, but we have added this ability nonetheless. This provides flexibility in case some users want to customize the translation process for their specific dialect types.

Grouping and Summaries Improvements and Fixes

SlickGrid (including our customized SleekGrid) offers remarkable flexibility and extensibility. It seamlessly works with various data sources, whether in-memory or remote, and supports easy extension through a straightforward plugin interface.

For example, the core SlickGrid components (slick.core.js and slick.grid.js) lack knowledge of grouping, group rows, or summaries. These functionalities are entirely managed by the data source or view.

In the context of SlickGrid's data view (known as RemoteView in Serenity), handling grouping and summary calculations is vital. Subsequently, a plugin is necessary to ensure that group rows are displayed correctly, complete with group titles, values, and item counts, along with footer sections containing calculated summaries. This essential plugin is called the GroupItemMetadataProvider (slick.groupitemmetadataprovider.js).

However, it's crucial to note that the GroupItemMetadataProvider is not a typical SlickGrid plugin. It must be registered both within the grid and the view, as it facilitates the coordination of metadata and operations between them.

It's worth mentioning that even though Serenity grids (DataGrids) typically feature a one-to-one relationship between a data source and its grid, it's usually possible to utilize a single data source with multiple SlickGrid instances.

In a basic grouping example from our common-features repository, you would find something like this:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    protected createSlickGrid() {
        var grid = super.createSlickGrid();

        // To enable grouping, this plugin must be registered
        grid.registerPlugin(new GroupItemMetadataProvider());

        //...
    }
}

However, there's a drawback to the above sample. Since the plugin is not registered in the data view (RemoteView), it remains unaware of the existing GroupItemMetadataProvider. Consequently, it creates a new one during the initialization of grouping:

function setGrouping(groupingInfo) {
    if (!options.groupItemMetadataProvider) {
        options.groupItemMetadataProvider = new Slick.Data.GroupItemMetadataProvider();
    }
    //...

Failure to create and register a GroupItemMetadataProvider as a plugin in the SlickGrid instance may result in group rows not being displayed correctly and can lead to errors.

These improvements and fixes are intended to enhance the grouping and summary features within SlickGrid for a more seamless and robust user experience.

Addressing Plugin Duplication Issue

The code block above is excerpted from the slick.dataview.js file within the original SlickGrid repository. A similar logic was reproduced in our custom RemoteView class.

When you do not pass the plugin to the view options, you end up with two instances of the same plugin coexisting. One instance is registered with the grid, while the other, though internally created by the data view, remains unregistered.

Initially, this situation was not a major concern, especially when the GroupItemMetadataProvider was created without any additional options. In such cases, the instance we created and the one created internally by the data view held the same set of options.

Due to this, even if we overlooked passing the GroupItemMetadataProvider to the data view, our sample continued to function as expected. However, issues arose after we introduced refinements to the GroupItemMetadataProvider class, leading to the problem detailed in the following GitHub issue:

To resolve this, it is imperative to structure the code as follows:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    private groupItemMetadataProvider: GroupItemMetadataProvider;

    protected getViewOptions() {
        var opt = super.getViewOptions();
        this.groupItemMetadataProvider = new GroupItemMetadataProvider({ 
            // Include any additional options you require
        });
        opt.groupItemMetadataProvider = this.groupItemMetadataProvider;
        return opt;
    }

    protected createSlickGrid() {
        var grid = super.createSlickGrid();
        grid.registerPlugin(this.groupItemMetadataProvider);
        //...
    }
}

By adopting this corrected code structure, you ensure that the issue of duplicated plugins is resolved, and your application works as intended.

Simplifying Plugin Registration

Previously, you needed to register the same plugin instance with both the grid and the view. However, with the latest SleekGrid (1.5.5+), you can access the GroupItemMetadataProvider instance directly from the view after it's created. This makes it unnecessary to maintain an extra private variable for the plugin instance.

Here's the updated approach:

export class GroupingAndSummariesInGrid extends EntityGrid<ProductRow, any> {

    protected getViewOptions() {
        var opt = super.getViewOptions();
        const groupItemMetadataProvider = new GroupItemMetadataProvider({ 
            // Include any additional options you require
        });
        opt.groupItemMetadataProvider = groupItemMetadataProvider;
        return opt;
    }

    protected createSlickGrid() {
        var grid = super.createSlickGrid();
        grid.registerPlugin(this.view.getGroupItemMetadataProvider());
        // Additional configuration...
    }
}

This streamlined code eliminates the need for an additional private variable to store the plugin instance.

In StartSharp, we've also enhanced our DraggableGroupingMixin to make it more intuitive. It now intelligently creates a GroupItemMetadataProvider if it's not already registered with the grid and also registers it with the view. This leads to a more concise, one-liner setup:

export class DragDropGroupingGrid extends OrderGrid {

    protected createToolbarExtensions() {
        super.createToolbarExtensions();

        new DraggableGroupingMixin({ grid: this });
    }
}

This modification automates the creation of a GroupItemMetadataProvider, its registration as a SlickGrid plugin, and the configuration of the groupItemMetaProvider for the data view.

Harmonizing HtmlEncode with AttrEncode (Deprecated)

We previously had two encoding functions: Q.htmlEncode and a similar function for attribute values called Q.attrEncode.

While attrEncode also escaped quote characters (" and '), htmlEncode did not, as they are not typically required to be escaped for HTML markup other than attribute values. Escaping angle brackets and ampersands was generally sufficient.

To avoid user confusion and given that there is no significant impact on performance when always escaping quotes, we have decided to make these functions work the same way.

Introducing Q.localText Method to Replace Q.text

We have renamed the text function to localText for ES module style TypeScript. The old function is still available but is now considered obsolete. Transitioning to localText() will make it easier to locate code blocks where local texts are used.

[WARNING] Potential Script Injection via the Translation Screen and Local Texts

When using local texts in a script method without calling htmlEncode (e.g., <div>{ text("SomeKey") }</div> instead of <div>{htmlEncode(text("SomeKey")) }</div>), an attacker could potentially inject JavaScript via the translations screen.

This issue affected parts of our demo since anyone could log in as the admin and access the translation screen. In our demo, translations are reset every 30 minutes, so the risk was limited. However, we strongly recommend reviewing your code, especially for standard screens like login, signup, forgot password, etc.

Please refer to our latest commits in the Serene/StartSharp/Common Features repositories for the fixes we applied. Even if only the admin can access the translations screen in your application, it's a good practice to mitigate risks, as trusting translators, even the admin, is not advisable.

We cannot default the "text," "localText," or "tryGetText" functions to htmlEncode, as they might be used in contexts other than HTML, such as functions expecting raw text, like notify messages, dialog titles, column titles, etc., which would result in double HTML encoding.

We recommend using the localText() function instead of the deprecated text() function for improved discoverability (but remember that it does not perform escaping).