Steps to configure a Liferay multi-tenancy environment
If you’ve read my previous article Liferay Saas solution – handling multi-tenancy where I describe the principles used to handle a multi-tenancy installation for Liferay, you are probably wondering about the technical details of how to actually configure a Liferay multi-tenancy environment. Well, here is a description of what we did. Hope you find it useful. If you have any questions please leave a comment an I will try to help.
In short these are the modifications made to allow Liferay to lazy connect to the shards and allow a fast start-up no matter the number of shards involved:
- Explicitly set the Hibernate dialect – this is done in the portal-ext.properties file;
- Database connection information and data source settings – dynamically create data sources and make necessary adjustments such that actual connection is delayed as much as possible
- Handling the Hibernate session factory – initialize Hibernate configuration only once, not for each shard
- Prevent start up initializations
- Initializations not made on start up should be made on first access of each shard / instance
- Initializations needed for Control Panel functions
1. Explicitly set the Hibernate dialect
[LIFERAY-HOMEDIR]/portal-ext.properties
This file is actually overwrites the default settings of the Portal. The default Portal settings are stored in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/portal-imp.jar. After following the documentation of Liferay this file looked somthing like
spring.configs=\
META-INF/base-spring.xml,\
\
META-INF/hibernate-spring.xml,\
META-INF/infrastructure-spring.xml,\
META-INF/management-spring.xml,\
\
META-INF/util-spring.xml,\
\
META-INF/jpa-spring.xml,\
\
META-INF/audit-spring.xml,\
META-INF/cluster-spring.xml,\
META-INF/editor-spring.xml,\
#META-INF/jcr-spring.xml,\
portal-shards-jcr.xml,\
META-INF/ldap-spring.xml,\
META-INF/messaging-core-spring.xml,\
META-INF/messaging-misc-spring.xml,\
META-INF/poller-spring.xml,\
META-INF/rules-spring.xml,\
META-INF/scheduler-spring.xml,\
META-INF/scripting-spring.xml,\
META-INF/search-spring.xml,\
META-INF/workflow-spring.xml,\
\
META-INF/counter-spring.xml,\
META-INF/document-library-spring.xml,\
META-INF/mail-spring.xml,\
META-INF/portal-spring.xml,\
META-INF/portlet-container-spring.xml,\
\
META-INF/ext-spring.xml,\
\
portal-shards.xml
shard.selector=com.liferay.portal.dao.shard.ManualShardSelector
shard.available.names=default,c1,c2,........,c15000
hibernate.dialect=org.hibernate.dialect.MySQLDialectIn this file we have placed the sharding settings according to the Liferay documentation. We have also added the hibernate.dialect explicitly as this was one of the reasons for which the portal tried to connect to each and every shard in order to detect the Hibernate dialect to use.
Also, notice that we have added a reference to portal-shards.xml which is actually a copy of the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/lib/portal-impl.jar/META-INF/shard-data-source-spring.xml. We did this so that we can modify the shards configuration without having to change the content of the jar everytime. Instead now, the portal looks for the file in [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/classes/.
To keep things clearer can add this to the file
include-and-override=portal-ext-extra-shards.properties
What this does is indicate to Liferay that it should load the file portal-ext-extra-shards.properties along side the portal-ext.properties. This allows us to separately store settings related to the sharding.
2. Database connection information and data source settings
One important thing to note is that for sharding one needs to add connection information in the portal-ext.properties file in the form of
jdbc.[shard-name].driverClassName=com.mysql.jdbc.Driver jdbc.[shard-name].url=[Connection string] ...
This information is used to create the DataSource objects that the application uses. The DataSource objects are defined in the Spring context, namely in the portal-shards.xml file created as indicated above. This instantiates a few singletons org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy with a target data source set as com.liferay.portal.dao.jdbc.util.DataSourceFactoryBean which in turn has as a parameter named parameterPrefix indicating the prefix used to retrieve the connection information from the portal-ext.properties file. LazyConnectionDataSourceProxy is actually a very useful choice in our situation as this type of data source will actually connect to the database only upon the first creation of a statement. So far we did not change anything from the default configuration, but to support the 15000 clients / portal instances we need to add 15000 connection information groups in the portal-ext.properties file and 15000 Spring beans in the portal-shards.xml file. This can be done by generating these entries with some script, but one thing we chose to do is to aviod overfilling the portal-ext.properties file which is more often used for changing portal configurations and what we did, we chose to replace the default com.liferay.portal.dao.jdbc.util.DataSourceFactoryBean from portal-shards.xml with our own implementation which does not look at the 15000 groups of connection information lines, but dynamically generates connections based on some rules for schema names, database user names and passwords.
In short we created a org.springframework.beans.factory.config.AbstractFactoryBean and put it in place of the com.liferay.portal.dao.jdbc.util.DataSourceFactoryBean. Our class createInstance creates one com.mchange.v2.c3p0.ComboPooledDataSource object for each entry in the Spring context and uses the propertyPrefix value to create the connection information. As we can see we used C3P0 as our connection pool provider as this is the default provider used by the Portal.
IMPORTANT NOTE: Through testing we noted that even though the portal uses LazyConnectionDataSourceProxy which only initializes the connection pool upon first creating a statement, when we started the Portal it was creating connections thus slowing down the start up time. This was because LazyConnectionDataSourceProxy will try to connect on initialization if the defaultAutoCommit and defaultTransactionIsolation properties are not set. So we have set them such.
<bean id="shardDataSource0" lazy-init="true">
<property name="defaultAutoCommit" value="true"/>
<property name="defaultTransactionIsolation" value="4"/>
<property name="targetDataSource">
<bean lazy-init="true">
<property name="propertyPrefix" value="jdbc.default."/>
</bean>
</property>
</bean>3. Handling the Hibernate session factory
In the Portal shard configuration we notice that the Hibernate SessionFactory used is com.liferay.portal.dao.shard.ShardSessionFactoryTargetSource which in turn uses a key/value map with one com.liferay.portal.spring.hibernate.PortalHibernateConfiguration object for each shard. The latter is a LocalSessionFactoryBean responsible for the Hibernate configurations and mappings. This is all fine as is, but again, for 15000 it’s not ideal as for every shard the system will load the Hibernate configuration files 15000 times which means a long start up time.
What we did to avoid this is extend the com.liferay.portal.spring.hibernate.PortalHibernateConfiguration and override the newConfiguration method to only create the Configuration object once. This means that the Portal Hibernate mappings files are only loaded once. Through testing we also noted that the Portal still wanted to connect to each shard upon start up and found that it was due to Hibernate that needed to access the database metadata. There is a hibernate property that determines this and that is hibernate.temp.use_jdbc_metadata_defaults which defaults to true. So, we set this property to false in our version of the PortalHibernateConfiguration class.
Also in this PortalHibernateConfiguration we noticed that the buildSessionFactory method was called for each shard so we also made an override of that as the SessionFactory objects created were the same excepting the data source. We therefore created a method that creates a SessionFactory and sets a data source. Please see 6. Initializations needed for Control Panel functions where we used this method.
Here’s how our PortalHibernateConfiguration version looks like
..... extends com.liferay.portal.spring.hibernate.PortalHibernateConfiguration {
private static Configuration _config = null;
private static SessionFactory _sessionFactory = null;
@Override
protected Configuration newConfiguration() {
if (_config == null) {
_config = super.newConfiguration();
_config.getProperties().setProperty("hibernate.temp.use_jdbc_metadata_defaults", "false");
}
return _config;
}
@Override
protected SessionFactory buildSessionFactory() throws Exception {
if (_sessionFactory == null) {
_sessionFactory = super.buildSessionFactory();
}
return _sessionFactory;
}
public SessionFactory buildSessionFactory(DataSource dataSource) throws Exception {
this.setDataSource(dataSource);
return super.buildSessionFactory();
}Don’t forget replace all occurrences of com.liferay.portal.spring.hibernate.PortalHibernateConfiguration with your implementation in the portal-shards.xml. Also, place your implementation in a .jar file and drop it in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/lib folder.
4. Prevent start up initializations
Up to this point we have configured the Liferay Spring context to delay and avoid connecting to all the shards on context initialization with the though in mind that only when a user actually accesses their shard / instance the connection will actually be made.
Also through testing we realized that this was not enough, because the Portal does some initializations upon start up which involve accessing data form all shards. So, we had to find a way to prevent those initializations to take place before the Portal actually finished starting. Step by step, we were able to identify the initialization tasks that were performed on start up and tried to disable them. The idea was actually to disable them on start up, but perform them at a later time – on user access to the shard.
We identified that the initialization of each shard / instance was made inside the com.liferay.portal.servlet.MainServlet servlet on the initCompanies method, so we decided to override this method to only init the main shard. Here’s how our implementation looks like
... extends MainServlet {
@Override
protected void initCompanies() throws Exception {
ServletContext servletContext = getServletContext();
String[] webIds = PortalInstances.getWebIds();
PortalInstances.initCompany(servletContext, webIds[0]);
}
}In order for the new servlet to be used you have to replace com.liferay.portal.servlet.MainServlet with your implementation in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/web.xml file.
If everything was set up right then the Portal should start with this configuration in a reasonable amount of time (for me it takes around 80 seconds).
IMPORTANT NOTE: Though the Portal now starts quickly only the default shard will function properly. Remember that we have deactivated the start up initialization. We still have to initialize each shard upon access.
5. Initializations not made on start up should be made on first access of each shard / instance
In the previous step we have replaced the initialization of all the shards with the initialization of the default shard only. We would now like to have this initialization taking place upon first access of a shard. In order to do that we have decided to put the initialization code inside one of the filters of the Portal ROOT webapp. Namely, we have overridden the processFilter method of the com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter class. This filter is used to provide virtual host functionality, that is when a user accesses a certain virtual host pointing to the Liferay portal server it will route the user to the correct shard / instance corresponding to the virtual host name. This filter was appropriate for our goals and also it is the first filter in the list of filters giving it priority over the others.
In our implementation of the filter we identify the shard / instance accessed by the name of the virtual host, we call the initialization code for a shard / instance (just like in the original com.liferay.portal.servlet.MainServlet.initCompanies() method). We also store the hosts for which the initialization has been made such that we only initialize one shard upon first access and not every time.
The resulting code for the filter looks something like this
... extends VirtualHostFilter {
private static List lstInitializedHosts = new ArrayList();
@Override
protected void processFilter(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws Exception {
String host = PortalUtil.getHost(request);
if (!lstInitializedHosts.contains(host)) {
lstInitializedHosts.add(host);
Company company = CompanyLocalServiceUtil.getCompanyByVirtualHost(host);
PortalInstances.initCompany(request.getSession().getServletContext(), company.getWebId());
}
super.processFilter(request, response, filterChain);
}
}Again, in order for the new filter to be used you have to replace com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter with your implementation in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/web.xml file.
Place your implementation in a .jar file (can be the same as for the servlet) and drop it in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/lib folder.
With these settings the Portal will start just as quick as with only one shard / instance and upon each access of an existing shard it will create a database connection and then initialize the shard correctly such that everything will work as expected.
6. Initializations needed for Control Panel functions
Even though the Portal shards / instances work correctly, if you go to the Control Panel and try to define a new instance, the Portal will start to give errors. This is because as you remember each instance is only initialized when directly accessed by a user through a virtual host name. So why does it give an error, because in the default shard’s Control Panel things don’t stay local only on the current shard, but the Portal accesses all sorts of other information from other shards. Those shards might not be yet initialized and therefore the error. Actually the error had to do with the fact that the SessionFactory was not correctly built.
To fix this problem we have turned to the Aspect Oriented configuration used by the portal. What this does is wrap around Portal API calls such that it detects to which shard / instance the call refers to and sets creates the conditions for the call to get executed on the correct shard. We decided that this might be a good place to detect if calls to other instances than the default instance are made such that we initialize them before making the call so that the errors do not appear anymore.
We have extended the ShardAdvice class of the portal in this way
... extends ShardAdvice {
private static List lstInitializedShards = new ArrayList();
private static ShardSelector _shardSelector;
static {
try {
_shardSelector = (ShardSelector) Class.forName(PropsValues.SHARD_SELECTOR).newInstance();
} catch (Exception e) {
}
}
/**
* @param shardName
* @throws Exception
*/
@SuppressWarnings("unchecked")
private void buildSessionFactory(String shardName, String methodCall) throws Exception {
if (PropsValues.SHARD_DEFAULT_NAME.equals(shardName) || lstInitializedShards.contains(shardName)) {
return;
}
lstInitializedShards.add(shardName);
ShardSessionFactoryTargetSource shardSessionFactoryTargetSource =
(ShardSessionFactoryTargetSource) PortalBeanLocatorUtil.locate("shardSessionFactoryTargetSource");
Class cls = ShardSessionFactoryTargetSource.class;
Field fld = cls.getDeclaredField("_sessionFactories");
fld.setAccessible(true);
Map _sessionFactories = (Map) fld
.get(shardSessionFactoryTargetSource);
int dataSourceIndex = 0;
if (!"default".equals(shardName)) {
dataSourceIndex =
Integer.parseInt(StringUtils.replace(StringUtils.replace(shardName, "c", ""), ".", ""));
}
DataSource dataSource = (DataSource) PortalBeanLocatorUtil.locate("shardDataSource" + dataSourceIndex);
PortalHibernateConfiguration sessionFactory = new PortalHibernateConfiguration();
_sessionFactories.put(shardName, sessionFactory.buildSessionFactory(dataSource));
}
@Override
public Object invokeByParameter(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
String methodSignature = _getSignature(proceedingJoinPoint);
if (!"ResourceCodeLocalServiceImpl.checkResourceCodes()".equals(methodSignature)) {
Object[] arguments = proceedingJoinPoint.getArgs();
long companyId = (Long) arguments[0];
Shard shard = ShardLocalServiceUtil.getShard(Company.class.getName(), companyId);
String shardName = shard.getName();
buildSessionFactory(shardName, "invokeByParameter: " + methodSignature);
}
return super.invokeByParameter(proceedingJoinPoint);
}
@Override
public Object invokeCompanyService(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
String methodName = proceedingJoinPoint.getSignature().getName();
Object[] arguments = proceedingJoinPoint.getArgs();
String shardName = PropsValues.SHARD_DEFAULT_NAME;
if (methodName.equals("addCompany")) {
String webId = (String) arguments[0];
String virtualHost = (String) arguments[1];
String mx = (String) arguments[2];
shardName = (String) arguments[3];
shardName = _getCompanyShardName(webId, virtualHost, mx, shardName);
} else if (methodName.equals("checkCompany")) {
String webId = (String) arguments[0];
if (!webId.equals(PropsValues.COMPANY_DEFAULT_WEB_ID)) {
if (arguments.length == 3) {
String mx = (String) arguments[1];
shardName = (String) arguments[2];
shardName = _getCompanyShardName(webId, null, mx, shardName);
}
try {
Company company = CompanyLocalServiceUtil.getCompanyByWebId(webId);
shardName = company.getShardName();
} catch (NoSuchCompanyException nsce) {
}
}
} else if (methodName.startsWith("update")) {
long companyId = (Long) arguments[0];
Shard shard = ShardLocalServiceUtil.getShard(Company.class.getName(), companyId);
shardName = shard.getName();
}
buildSessionFactory(shardName, "invokeCompanyService: " + _getSignature(proceedingJoinPoint));
return super.invokeCompanyService(proceedingJoinPoint);
}
private String _getCompanyShardName(String webId, String virtualHost, String mx, String shardName) {
Map shardParams = new HashMap();
shardParams.put("webId", webId);
shardParams.put("mx", mx);
if (virtualHost != null) {
shardParams.put("virtualHost", virtualHost);
}
shardName = _shardSelector.getShardName(ShardSelector.COMPANY_SCOPE, shardName, shardParams);
return shardName;
}
private String _getSignature(ProceedingJoinPoint proceedingJoinPoint) {
String methodName = StringUtil.extractLast(proceedingJoinPoint.getTarget().getClass().getName(),
StringPool.PERIOD);
methodName += StringPool.PERIOD + proceedingJoinPoint.getSignature().getName() + "()";
return methodName;
}
}What the code does is build SessionFactory objects with the right data source and set them in the right place. Please note that the code relies on the presumption that the shards are identified as default, c1, c2, c3, …
Place your implementation in a .jar file (can be the same as for the servlet and filter) and drop it in the [LIFERAY-HOMEDIR]/tomcat/webapps/ROOT/WEB-INF/lib folder.
In order for your implementation to be used by the Portal you have to replace the entry in the portal-shards.xml
replace this
<bean id="com.liferay.portal.dao.shard.ShardAdvice">
with this
<bean id="your.implementation.of.ShardAdvice">