2.3.24 Release Candidate 1

Date of release for Release Candidate 1: 2015-01-12

Planned final release date: around 2016-02-20

Please test for backward compatibility, and help polishing the new features!

Legal changes

The owner of FreeMarker is now the Apache Software Foundation. The license is still Apache License Version 2.0, just like earlier, but the owner is different. The official full product name has changed to Apache FreeMarker.

Disclaimer: Apache FreeMarker is an effort undergoing incubation at The Apache Software Foundation (ASF). Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

Changes on the FTL side

  • The most important new feature of this release is the auto-escaping and output formats mechanism, which deprecates escaping with the escape directive. These are the items related to this new mechanism (see early linked section for a guide):

    • New ftl header options, ouput_format and auto_esc to override the output_format and auto_escaping settings of the template, like <#ftl output_format='HTML' auto_esc=false>.

    • New built-in: no_esc. Used to prevent auto-escaping, like ${descriptionInHtml?no_esc}. This doesn't work with <#escape ...>, only with the new auto-escaping mechanism.

    • New FTL type, "markup output". This is somewhat similar to string, but values of this type aren't auto-escaped by the new escaping mechanism, because they are known to already hold markup. (For example, ?esc and ?no_esc returns a value of this type, hence their results are protected from double-escaping problems.)

    • New built-in: esc. Used for escaping with the current output format when auto-escaping is disabled.

    • Added new built-in: markup_string. This returns the markup of a markup output value as a string.

    • Added new built-in: is_markup_output, returns true if the value is of type "markup output".

    • New directive: outputformat, used to change the output format for a section of a template, like <#outputformat "XML">...</#outputformat>

    • New directives: noautoesc and autoesc, used to turn auto-escaping off and on for a section of a template, like <#noautoesc>...</#noautoesc>.

    • New built-in variable, .output_format, returns the current output format at the place of invocation.

    • New built-in variable, .auto_esc, returns the boolean that tells if auto-escaping is active at the place of invocation.

    • Block assignments, like <#assign captured>...</#assign>, when the current output_format is some kind of markup (like HTML), will store the captured output not with string type, but with "markup output" type. Thus ${captured} will not get unwanted escaping.

    • The + operator (concatenation) works with the new "markup output" type as well. Like someMarkup + somePlainText will result in markup where somePlainText is included properly escaped.

    • The has_content built-in now supports markup output values, considering 0 length markup as empty.

  • There can be now user defined number and date/time/datetime formatters, defined by the programmers when configuring FreeMarker, which can be referred with strings starting with "@", like in <#setting number_format='@foo'>, or ${n?string.@foo_params}, <#setting number_format='@foo params'>, or ${n?string.@foo}, ${n?string.@foo_params}. For backward compatibility, the initial @ only has this special meaning if either you have any custom formats or the incompatible_improvements setting is at lest 2.3.24.

  • Everywhere where Java DecimalFormat patterns are used (like in ?string('0.##') or <#setting number_format="0.##">), now it's possible to specify options like rounding mode or the symbols used, with a FreeMarker-specific extension to the pattern syntax.

  • Added new special variable: .incompatible_improvements, which returns the incompatbile_improvements setting of the current FreeMarker configuration, as a string.

  • ?date, ?time and ?datetime now can be called as 0 argument method, like ?date(), etc., which returns the exact object that TemplateDateFormat.parse returns, instead of the tricky multi-type object that just using ?date returns. Because custom TemplateDateFormat implementations may return custom TemplateDateModel implementations, keeping the exact class can be important in some applications.

  • <@ and </@ is now allowed in string literals that contain ${exp}, and will be part of the literal as is. Earlier it was a syntactical error.

  • Bug fixed: With incompatible_improvements set to 2.3.24 (see how here...), m?is_sequence doesn't return true for Java methods wrapped by BeansWrapper and its subclasses (most notably DefaultObjectWrapper) anymore, as they only implement the [index] operator, but not ?size, which causes <#list ...> to fail among others. (They shouldn't implement either, but this is historical heritage.)

Changes on the Java side

  • Attention! FreeMarker now requires at least 1.5 (aka. Java 5). 2.3.24 has only required Java 1.4. (Reason: Without this, new public API-s couldn't use generics, which affect negatively the majority of users, while old installations that are still using 1.4 are unlikely to update FreeMarker anyway.)

  • Attention! FreeMarker's JSP support (if it's used) now requires at least JSP 2.0. Earlier it only required JSP 1.1. (Reason: The jsp-api dependency for JSP 1.x, which was needed for building, can't be legally present in the Maven Central Repository, nor be provided by freemarker.org.)

  • Added new configuration setting: template_configurations. This allows overriding the settings coming from the shared Configuration object for individual templates, based on template name patterns. See more here...

  • Related to the new auto-escaping mechanism:

    • Added new configuration setting: recognize_standard_file_extensions. When true, templates whose source name ends with ".ftlh" will get HTML output_format, and those whose name ends with ".ftlx" get XML output_format, in both cases with auto-escaping on. If the incompatible_improvements setting is set to 2.3.24 (or higher) then this setting defaults to true. Otherwise it's false, but enabling it is highly recommended.

    • Added new configuration setting: output_format. This specifies the name of the output format (such as HTML, XML, plainText, etc.) that governs auto-escaping. The output format can be different for different templates, using the template_configurations setting (see here how...).

    • Added new configuration setting: registered_custom_output_formats. With this you can add new OutputFormat-s that templates can refer to by name (like in <#ftl output_format="foo">).

    • Added new configuration setting: auto_escaping_policy. This decides when auto-escaping should be enabled depending on the current output format.

  • Changes related to the custom number and date/time/datetime formating feature:

    • Added new classes for implementing custom formatters: freemarker.core.TemplateNumberFormat, TemplateNumberFormatFactory, TemplateDateFormat, TemplateDateFormatFactory, also the exceptions these can throw. These allow implementing any kind of formatting rules that are doable in Java (they aren't restricted to any java.text formatters). Furthermore these formatters get the TemplateModel instead of a the bare java.lang.Number or java.util.Date, which let you use the extra application-specific meta information that you may pack into the TemplateModel-s, such as physical unit, preferred precision, and so on.

    • Added custom_number_formats and custom_date_formats settings (Configurable.setCustomNumberFormats(Map<String, TemplateNumberFormatFactory>) and Configurable.setCustomDateFormats(Map<String, TemplateDateFormatFactory>)) with which you can register your own formats. If you set incompatible_improvements to 2.3.24, and only then, these formats can be referred from everywhere where you can use a string to define a format (often as a pattern), with a format string like "@foo" or "@foo params", where "foo" is the key in the Map<String, ...> parameter of the earlier shown methods, and the parameters (if any) are interpreted by the TemplateXxxFormatFactory. Like, you can do cfg.setNumberFormat("@foo params"), or <#setting number_format='@foo params'>, or ${n?string.@foo_params}. For backward compatibility, the initial @ only has this special meaning if either you have any custom formats or the incompatible_improvements setting is at least 2.3.24. Note that the custom_number_formats and custom_date_formats settings can be set per-template (via the new template_configurations settings) or per-Environment too, thus @foo can mean something different in different templates.

    • Added new Environment methods returning TemplateNumberFormat and TemplateDateFormat objects. See getTemplateNumberFormat(...) and getTemplateDateFormat(...) variations.

    • Added freemarker.core.AliasTemplateNumberFormatFactory and AliasTemplateDateFormatFactory, which can be used to create custom formats that are aliases to other formats. For example, instead of writing ${n?string["0.00"]} again and again, you can define the custom format "price" as the alias to the format string "0.00" in the configuration, and then use ${n?string.@price}. Thus, you can control at a central place how prices look. Furthermore, the alias can chose a different target format string depending on the current locale; this is especially useful for dates, where conventions can significantly differ in different countries.

    • It's now possible to have HTML or other markup in number and date/time/datetime formatting results, like 1.23*10<sup>6</sup>, which won't be accidentally auto-escaped, as FreeMarker knows that it's already HTML. (See [TODO] as an example.) Note that no out-of-the-box format formats to markup (at the moment), but you could write such custom format.

    • The internal format object caching architecture has been reworked, so that it can handle custom formats too. But this reworking also fixes some bottlenecks under highly concurrent load, and some (otherwise unlikely) memory leaking possibilities.

  • In the number_format configuration setting, when it holds a Java DecimalFormat pattern (like "0.##"), it's now possible to specify options like rounding mode or the symbols used, with a FreeMarker-specific extension to the pattern syntax.

  • New FreemarkerServlet init-params (see the FreemarkerSerlvet API documentation for details):

    • OverrideResponseContentType: Specifies when should we override the contentType that's already set (i.e., non-null) in the HttpServletResponse. Earlier, we have always set it, and that's still the default behavior. But now that this init-param exists, you can change that behavior, so that the contentType you have specified before forwarding to FreemarkerServlet matters.

    • OverrideResponseLocale: Specifies if should we override the contentType that's already set (i.e., non-null) in the HttpServletResponse. Earlier, we have always set it, but now this behavior can be changed so that we only set it if it wasn't already set.

    • ResponseCharacterEncoding: Deprecates the old (and quirky) logic of specifying the output charset, which was putting it into the ContentType init-param after the MIME type, otherwise falling back to using the template file charset. The possible values are legacy (the default for backward compatibility), fromTemplate (which is legacy without quirks, and is aware of the outputEncoding setting), doNotSet (keeps what the caller has already set in the ServletRespone) and force followed by a charset name (forces a specific output charset).

  • Added freemarker.cache.ByteArrayTemplateLoader, which is similar to StringTemplateLoader, but stores the templates as byte[]-s instead of as String-s.

  • Upgraded JavaCC (used during build to generate the FTL parser) from 3.2 to 6.1.2.

  • Added Configurable.getSettingNames(boolean camelCase), which returns the set of valid setting names. This can be useful for auto-completion and such.

  • Fixes and improvements in the "object builder" mini-language used for configuring FreeMarker from java.util.Properties or other string-only sources (not used in templates). This is not to be confused with the template language syntax, which has nothing to do with the "object builder" syntax we are writing about here. The improvements are:

    • For nested builder expressions, the top-level result class restriction were applied accidentally.

    • When resolving an expression like com.example.Foo(), if there's a builder class (com.example.FooBuilder), the non-builder class (com.example.Foo) need not exist anymore. After all, FooBuilder.build() instantiates from any class it wants to anyway.

    • TimeZone objects can be created like TimeZone("UTC"), despite that there's no a such constructor.

    • Added support for creating lists with [ item1, item2, ... itemN ] syntax.

    • Added support for creating maps with { key1: value1, key2: value2, ... keyN: valueN } syntax.

    • Number without decimal point will now be parsed to Integer, Long, or BigInteger, depending in the size of the number. Earlier all number were BigDecimal-s, but it had little importance as number types are converted to the constructor parameter type anyway.

    • Number literals can have Java type specified postfixes (f, d, l), plus bd for BigDecimal and bi for BigInteger.

    • Public static fields can be referred, like com.example.MyClass.MY_CONSTANT or Configuration.AUTO_DETECT_TAG_SYNTAX.

  • Decreased the stack usage of template execution, which can have importance if you have very very deeply nested templates.

  • Bug fixed, only with incompatible_improvements set to 2.3.24 (see how here...): Expressions inside interpolations that were inside string literal expressions (not ${...}-s in general), like in <#assign s="Hello ${name}!">, always used incompatibleImprovements 0 (2.3.0 in effect). This means that expressions inside string literals had missed the ?html, ?iso_..., ?is_enumerable, ?c, etc. fixes/improvements.

  • Bug fixed [439]: FileTemplateLoader with emulateCaseSensitiveFileSystem set to true (used for development) wasn't properly synchronized, leading to random NullPointerException-s or other misbehavior.

  • Bug fixed: It wasn't well defined when a Java Iterator counts as empty. Depending on what ObjectWrapper you are using, one of these fixes apply:

    • DefaultObjectWrapper (fix is always active): Operations on the Iterator that only check if it's empty without reading an element from it, such as ?has_content, won't cause the a later iteration (or further emptiness check) to fail anymore. Earlier, in certain situations, the second operation has failed saying that the iterator "can be listed only once".

    • BeansWrapper (when it's not extended by DefaultObjectWrapper), if it's incompatibleImprovements property is set to 2.3.24 (or higher): Iterator-s were always said to be non-empty when using ?has_content and such (i.e., operators that check emptiness without reading any elements). Now an Iterator counts as empty exactly if it has no elements left. (Note that this bug has never affected basic functionality, like <#list ...>.)

  • Bug fixed: The (rarely used) cause exception of ParseException-s wasn't set

  • Bug fixed: When the incomaptible_improvements setting of an existing Configuration was changed, the template cache sometimes wasn't recreated, hence old templates could survive.

  • Bug fixed, with incompatible_improvements set to 2.3.24 (see how here...): The #import directive meant to copy the library variable into a global variable if it's executed in the main namespace, but that haven't happened when the imported template was already imported earlier in another namespace.

  • Fixes in the XML processing feature (freemarker.ext.dom):

    • Bug fixed: XPath queries that has only contained characters that are valid in XML element names and has also contained :: (which is valid in names in namespace-unware documents), like e['following-sibling::foo'], were interpreted as literal element names (giving 0 hits) rather than as XPath expressions. Note that there were no such problem with e['following-sibling::*'] for example, as it's not a valid XML element name according the XML specification. This fix can actually break applications that has processed namespace unaware XML that use :: as part of element or attribute names, but such an application is highly unlikely, unlike running into the fixed problem. (Unfortunately, using incompatible_improvements wasn't technically possible here.)

    • Bug fixed: The @@qname of elements that belong to the XML namespace declared as the default via <#ftl ns_prefixes={'D':'...', ... }> no longer starts with D:, instead they just start with no name space prefix.

    • Bug fixed: In the markup returned by the @@markup key, when there were multiple namespaces for which there was no prefix associated with via <#ftl ns_prefixes=...>, all those namespaces were assigned to the same auto-generated xmlns prefix (usually "a"). Now they will get "a", "b", "c", etc. prefixes.

  • Internal code cleanup: Mostly for consistent source code formatting, also many parser construction/setup cleanup

  • JSP TLD loading now quotes the location of jar-s (and other zip-s) which can't be loaded due to zip format errors in the error message.

  • Added an overload to Configuration.getSupportedBuiltInNames and Configuration.getSupportedBuiltInDirectiveNames that has a namingConvention parameter. This is useful for tooling as since 2.3.23 we support both camel case naming convention (like s?upperCase) and the legacy one (like s?upper_case). Furthermore the old 0 argument overload will now utilize Configuration.getNamingConvention() to only return the relevant names if it's not AUTO_DETECT_NAMING_CONVENTION.

  • Internal reworking to simplify the AST (the TemplateElement structure). The related technically public API was marked as internal for a good while. For those who still use that API, the visible change is that TemplateElement-s now almost never has a MixedContent parent, instead, the parent is directly whatever element the child element indeed belongs under when you look at the source code (like the enclosing #list for example, while earlier you often had to go through a MixedContent first whose parent was the #list). Note that when you have moved downwards, i.e., towards the child elements, these MixedContent parents weren't visible and were silently skipped, so the tree traversal API was inconsistent. Now it's consistent.

  • The non-public AST API of freemarker.core.StringLiteral-s has been changed. In principle it doesn't mater as it isn't a public API, but some might used these regardless to introspect templates. Earlier it had an "embedded template" parameter inside, now it has 0 (for purely static string literals), one or more "value part"-s, which are String-s and Interpolation-s.

Changes compared to 2.3.24 Preview 1

  • Environment now has public TemplateDateFormat returning methods.

  • The various TemplateXxxFormat and TemplateXxxFormatFacotry exceptions were united under a common abstract superclass, TemplateValueFormatException, which is now what the methods of said classes throw (mostly).

  • Added freemarker.core.AliasTemplateNumberFormatFactory and AliasTemplateDateFormatFactory, which can be used to create custom formats that are aliases to other formats. For example, instead of writing ${n?string["0.00"]} again and again, you can define the custom format "price" as the alias to the format string "0.00" in the configuration, and then use ${n?string.@price}. Thus, you can control at a central place how prices look. Furthermore, the alias can chose a different target format string depending on the current locale; this is especially useful for dates, where conventions can significantly differ in different countries.

  • Everywhere where Java DecimalFormat patterns are used (like in the number_format configuration setting, or in ?string('0.##')), now it's possible to specify options like rounding mode or the symbols used, with a FreeMarker-specific extension to the pattern syntax.

  • In number/date/time/datetime format strings, initial @ can be used to refer to custom formats even if incompatible_improvements is less than 2.3.24, as far as there's any custom format defined. As custom formats is also a 2.3.24 feature, it's backward compatible this way.

  • @@ can't be used to escape @ at the beginning of format strings anymore, because it was confusing on Android, where DecimalFormat supports patterns like "@@@@@@". If a pattern has to output a literal @ as the first character, you can simply use quoting as defined by DecimalFormat and SimpleDateFormat (for example, "'@'0.##").

  • ?date, ?time and ?datetime now can be called as 0 argument method, like ?date(), etc., which returns the exact object that TemplateDateFormat.parse returns, instead of the tricky multi-type object that just using ?date returns. Because custom TemplateDateFormat implementations may return custom TemplateDateModel implementations, keeping the exact class can be important in some applications.

  • The TemplateDateFormat.parse has been worked out (earlier it was a but different and was marked as draft). TemplateNumberFormat has also have a parse method now, though it's just a final placeholder now.

  • TemplateNumberFormat.format/TemplateDateFormat.format overloads were changed to formatToPlainText and format, the last returning either String or a TemplateMarkupOutputModel. Also, formatting to markup is now actually used by FreeMarker. Thus, it's now possible to have HTML or other markup in number and date/time/datetime formatting results, like 1.23*10<sup>6</sup>, which won't be accidentally auto-escaped, as FreeMarker knows that it's already HTML. (See [TODO] as an example.) Note that no out-of-the-box format formats to markup (at the moment), but you could write such custom format.

  • ${...} inside string literals is equivalent to using the + operator again. This rule was broken by + supporting markup operands, while ${...} inside string literals didn't. Now similarly as "foo " + someMarkup works and gives a markup result, "foo ${someMarkup}" does too.

  • The extended decimal format options don't use abbreviations anymore. Like instead of rnd=hu, now there's roundingMode=halfUp.

  • Added XHTMLOutputFormat and TemplateXHTMLOutputModel.

  • XMLOutputFormat now uses application/xml MIME type instead of text/xml MIME type

  • RTFOutputFormat now uses application/rtf MIME type instead of text/rtf

  • Added non-escaping formats (just for the MIME type): JavaScriptOutputFormat, JSONOutputFormat, CSSOutputFormat.

  • Added JavaScriptOutput

  • Added new built-in: is_markup_output, returns true if the value is of type "markup output".

  • New FreemarkerServlet init-params (see the FreemarkerSerlvet API documentation for details):

    • OverrideResponseContentType: Specifies when should we override the contentType that's already set (i.e., non-null) in the HttpServletResponse. Earlier, we have always set it, and that's still the default behavior. But now that this init-param exists, you can change that behavior, so that the contentType you have specified before forwarding to FreemarkerServlet matters.

    • OverrideResponseLocale: Specifies if should we override the contentType that's already set (i.e., non-null) in the HttpServletResponse. Earlier, we have always set it, but now this behavior can be changed so that we only set it if it wasn't already set.

    • ResponseCharacterEncoding: Deprecates the old (and quirky) logic of specifying the output charset, which was putting it into the ContentType init-param after the MIME type, otherwise falling back to using the template file charset. The possible values are legacy (the default for backward compatibility), fromTemplate (which is legacy without quirks, and is aware of the outputEncoding setting), doNotSet (keeps what the caller has already set in the ServletRespone) and any valid charset name (forces a specific output charset).

  • The new TemplateConfigurer class was renamed to TemplateConfiguration, and the related configuration setting from template_configurers to template_configurations. Also, the TemplateConfigurer.configure method was renamed to TemplateConfiguration.apply.

  • Bug fixed: It wasn't well defined when a Java Iterator counts as empty. Depending on what ObjectWrapper you are using, one of these fixes apply:

    • DefaultObjectWrapper (fix is always active): Operations on the Iterator that only check if it's empty without reading an element from it, such as ?has_content, won't cause the a later iteration (or further emptiness check) to fail anymore. Earlier, in certain situations, the second operation has failed saying that the iterator "can be listed only once".

    • BeansWrapper (when it's not extended by DefaultObjectWrapper), if it's incompatibleImprovements property is set to 2.3.24 (or higher): Iterator-s were always said to be non-empty when using ?has_content and such (i.e., operators that check emptiness without reading any elements). Now an Iterator counts as empty exactly if it has no elements left. (Note that this bug has never affected basic functionality, like <#list ...>.)

  • Fixes and improvements in the "object builder" syntax used for configuring FreeMarker from java.util.Properties (or other string-only sources). This is not to be confused with the template language syntax, which has nothing to do with the "object builder" syntax we are writing about here. The improvements are:

    • Number literals can have Java type specified postfixes (f, d, l), plus bd for BigDecimal and bi for BigInteger.

    • Public static fields can be referred, like com.example.MyClass.MY_CONSTANT or Configuration.AUTO_DETECT_TAG_SYNTAX.

  • The parser used by ?interpet and ?eval inherits not only the output_format of its surrounding lexical context, but also the auto-escaping policy of it (basically, if auto-escapig is on or off).

  • Bug fixed, with incompatible_improvements set to 2.3.24 (see how here...): The #import directive meant to copy the library variable into a global variable if it's executed in the main namespace, but that haven't happened when the imported template was already imported earlier in another namespace.

  • Fixes in the XML processing feature (freemarker.ext.dom):

    • Bug fixed: XPath queries that has only contained characters that are valid in XML element names and has also contained :: (which is valid in names in namespace-unware documents), like e['following-sibling::foo'], were interpreted as literal element names (giving 0 hits) rather than as XPath expressions. Note that there were no such problem with e['following-sibling::*'] for example, as it's not a valid XML element name according the XML specification. This fix can actually break applications that has processed namespace unaware XML that use :: as part of element or attribute names, but such an application is highly unlikely, unlike running into the fixed problem. (Unfortunately, using incompatible_improvements wasn't technically possible here.)

    • Bug fixed: The @@qname of elements that belong to the XML namespace declared as the default via <#ftl ns_prefixes={'D':'...', ... }> no longer starts with D:, instead they just start with no name space prefix.

    • Bug fixed: In the markup returned by the @@markup key, when there were multiple namespaces for which there was no prefix associated with via <#ftl ns_prefixes=...>, all those namespaces were assigned to the same auto-generated xmlns prefix (usually "a"). Now they will get "a", "b", "c", etc. prefixes.

  • Bug fixed: With incompatible_improvements set to 2.3.24 (see how here...), m?is_sequence doesn't return true for Java methods wrapped by BeansWrapper and its subclasses (most notably DefaultObjectWrapper) anymore, as they only implement the [index] operator, but not ?size, which causes <#list ...> to fail among others. (They shouldn't implement either, but this is historical heritage.)

  • Added an overload to Configuration.getSupportedBuiltInNames and Configuration.getSupportedBuiltInDirectiveNames that has a namingConvention parameter. This is useful for tooling as since 2.3.23 we support both camel case naming convention (like s?upperCase) and the legacy one (like s?upper_case). Furthermore the old 0 argument overload will now utilize Configuration.getNamingConvention() to only return the relevant names if it's not AUTO_DETECT_NAMING_CONVENTION.