-->
These old forums are deprecated now and set to read-only. We are waiting for you on our new forums!
More modern, Discourse-based and with GitHub/Google/Twitter authentication built-in.

All times are UTC - 5 hours [ DST ]



Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 10 posts ] 
Author Message
 Post subject: Hibernate second level cache consistency
PostPosted: Tue Oct 31, 2017 8:59 am 
Newbie

Joined: Mon Oct 30, 2017 8:02 am
Posts: 5
In our project we are using hibernate (4.3.11) and hazelcast 3.6.5 for distributed second lvl cache.

Here is sample test that explains problem we are facing:

Code:
@Test
public void cacheTest() {

    // Step 1: create account with someValue=111
    Account account = account().withName("test hz account").withSomeValue(111).build();
    account = restClient.upsertAccount(account);

    //Step 2: create user with ref to account
    User user = user().withAccount(account).build();
    restClient.upsertUser(user);

    //Step 3: update account someValue to 222
    account.setSomeValue(222);
    restClient.upsertAccount(account);

    //search users by account id
    List<User> users = restClient.searchUsersByAccountId(BRAND, account.getId());

    assertThat(users.size(), is(1));
    //assertion below fails, user.account.someValue=111
    assertThat(users.get(0).getAccount().getSomeValue(), is(222));
}


test is run on application with two nodes. Here is how requests are balanced to different nodes:

node1:
172.20.0.1 - - [12:30:46 +0200] "POST /accounts/BRAND/ HTTP/1.1" 200 321
172.20.0.1 - - [12:30:47 +0200] "POST /accounts/BRAND/ HTTP/1.1" 200 321
node2:
172.20.0.1 - - [12:30:47 +0200] "POST /users/BRAND/ HTTP/1.1" 200 542
172.20.0.1 - - [12:30:48 +0200] "POST /users/BRAND/search?start=0&limit=1 HTTP/1.1" 200 617

I have found that 2nd lvl cache contains two instances of same account. One with old field value 111 and one with new one 222.

First instance is added to cache on node1 on account insert, and second on node2 on user insert (we have account get call on user insert). As I understand these two account instances in cache causes inconsistency.

Any ideas why this could happen and how to debug this issue?

Here is data model:

Code:
@Getter
@Setter
@NoArgsConstructor
@Entity
@Table(name = "accounts")
@Cache(usage = READ_WRITE)
public class Account extends BrandIdEntity {
     @Column(name = "name")
     private String name;

     @Column(name = "some_value")
     private Integer someValue;
}

@Getter
@Setter
@NoArgsConstructor
@Entity
@Table(name = "users")
@Cache(usage = READ_WRITE)
public class User extends BrandIdEntity {

    @Column(name = "account_id")
    private String accountId;

    @ManyToOne(cascade = ALL)
    @JoinColumns({
        @JoinColumn(name = "brand", nullable = false, insertable = false, updatable = false),
        @JoinColumn(name = "account_id", nullable = false, insertable = false, updatable = false) })
    private Account account;
}

@EqualsAndHashCode(of = "pk", callSuper = false)
@MappedSuperclass
public abstract class BrandIdEntity extends AuditedEntity<BrandIdPK> {

    @EmbeddedId
    private final BrandIdPK pk;

    public BrandIdEntity() {
        super();
        pk = new BrandIdPK();
    }

    public BrandIdEntity(BrandType brand, String id, IdGenType idGenType) {
        super();
        pk = new BrandIdPK(brand, id, idGenType);
    }
}

@Embeddable
@Getter
@Setter
@EqualsAndHashCode(callSuper = false)
@NoArgsConstructor
public class BrandIdPK extends BaseObject {

    @Column(name = "id")
    private String id;

    @Column(name = "brand")
    private String brand;

    public BrandIdPK(BrandType brand, String id, IdGenType idGenType) {
        this.brand = brand.name();
        if (id != null) {
            this.id = id;
        }
        else {
            this.id = next(idGenType);
        }
    }
}


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Tue Oct 31, 2017 9:45 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
As explained in this article, Hibernate only defines the contract for the 2nd-level cache API.

It is up to the 2nd-level cache provider to implement it so that it guarantees consistency.

Therefore, only the Hazelcast team can provide you the right answer. Nevertheless, in a distributed system cache, strong consistency is very hard to achieve. Better use optimistic locking to prevent lost updates during writes, as long as you might read stale data during reads.


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Tue Oct 31, 2017 6:47 pm 
Newbie

Joined: Mon Oct 30, 2017 8:02 am
Posts: 5
i have digged into a little bit and found out that a problem is seems to be in CacheKey serialization on different nodes.

I see that all cache key fields:

private final Serializable key;
private final Type type;
private final String entityOrRoleName;
private final String tenantId;
private final int hashCode;

except type are the same on different nodes. This causes different cache key serialized data hash code on different nodes.


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Wed Nov 01, 2017 3:43 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
The Type represents the identifier type.

How are they any different? Give an example of what different Type instances you see on different nodes.


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Wed Nov 01, 2017 4:28 am 
Newbie

Joined: Mon Oct 30, 2017 8:02 am
Posts: 5
we are using composite primary key: BrandIdPK, and cache keys looks like:

node1:

Code:
cacheKey = {CacheKey@15202} "com.project.data.Account#BrandIdPK[id=1,brand=BRAND]"
key = {BrandIdPK@15175} "BrandIdPK[id=1,brand=BRAND]"
type = {ComponentType@15211}
  typeScope = {TypeFactory$TypeScopeImpl@15215}
   factory = {SessionFactoryImpl@15178}
    name = "sessionFName"
    uuid = "98790167-fd8c-40eb-b1a0-8f4fb0c0ea26"
    entityPersisters = {HashMap@15230}  size = 4
    classMetadata = {Collections$UnmodifiableMap@15237}  size = 4
    collectionPersisters = {HashMap@15242}  size = 3
    collectionMetadata = {Collections$UnmodifiableMap@15243}  size = 3
    collectionRolesByEntityParticipant = {Collections$UnmodifiableMap@15244}  size = 3
    identifierGenerators = {HashMap@15245}  size = 4
    namedQueryRepository = {NamedQueryRepository@15246}
    filters = {HashMap@15247}  size = 0
    fetchProfiles = {HashMap@15248}  size = 0
    imports = {HashMap@15249}  size = 8
    serviceRegistry = {SessionFactoryServiceRegistryImpl@15250}
    jdbcServices = {JdbcServicesImpl@15251}
    dialect = {MySQL5InnoDBDialect@15252} "org.hibernate.dialect.MySQL5InnoDBDialect"
    settings = {Settings@15253}
    properties = {Properties@15254}  size = 100
    schemaExport = null
    currentSessionContext = null
    sqlFunctionRegistry = {SQLFunctionRegistry@15255}
    observer = {SessionFactoryObserverChain@15256}
    entityNameResolvers = {ConcurrentHashMap@15257}  size = 0
    queryPlanCache = {QueryPlanCache@15258}
    cacheAccess = {CacheImpl@15259}
    isClosed = false
    typeResolver = {TypeResolver@15260}
    typeHelper = {TypeLocatorImpl@15261}
    transactionEnvironment = {TransactionEnvironmentImpl@15262}
    sessionFactoryOptions = {SessionFactoryImpl$1@15263}
    customEntityDirtinessStrategy = {SessionFactoryImpl$3@15264}
    currentTenantIdentifierResolver = null
  propertyNames = {String[2]@15216}
  propertyTypes = {Type[2]@15217}
  propertyNullability = {boolean[2]@15218}
  propertySpan = 2
  cascade = {CascadeStyle[2]@15219}
  joinedFetch = {FetchMode[2]@15220}
  isKey = true
  hasNotNullProperty = true
  entityMode = {EntityMode@15221} "pojo"
  componentTuplizer = {PojoComponentTuplizer@15222}
  canDoExtraction = null
entityOrRoleName = "com.project.data.Account"
tenantId = null
hashCode = 203299394


node2:
Code:
object = {CacheKey@15202} "com.project.data.Account#BrandIdPK[id=1,brand=BRAND]"
key = {BrandIdPK@15175} "BrandIdPK[id=1,brand=BRAND]"
type = {ComponentType@15211}
  typeScope = {TypeFactory$TypeScopeImpl@15215}
   factory = {SessionFactoryImpl@15178}
    name = "sessionFName"
    uuid = "98790167-fd8c-40eb-b1a0-8f4fb0c0ea26"
    entityPersisters = {HashMap@15230}  size = 4
    classMetadata = {Collections$UnmodifiableMap@15237}  size = 4
    collectionPersisters = {HashMap@15242}  size = 3
    collectionMetadata = {Collections$UnmodifiableMap@15243}  size = 3
    collectionRolesByEntityParticipant = {Collections$UnmodifiableMap@15244}  size = 3
    identifierGenerators = {HashMap@15245}  size = 4
    namedQueryRepository = {NamedQueryRepository@15246}
    filters = {HashMap@15247}  size = 0
    fetchProfiles = {HashMap@15248}  size = 0
    imports = {HashMap@15249}  size = 8
    serviceRegistry = {SessionFactoryServiceRegistryImpl@15250}
    jdbcServices = {JdbcServicesImpl@15251}
    dialect = {MySQL5InnoDBDialect@15252} "org.hibernate.dialect.MySQL5InnoDBDialect"
    settings = {Settings@15253}
    properties = {Properties@15254}  size = 100
    schemaExport = null
    currentSessionContext = null
    sqlFunctionRegistry = {SQLFunctionRegistry@15255}
    observer = {SessionFactoryObserverChain@15256}
    entityNameResolvers = {ConcurrentHashMap@15257}  size = 0
    queryPlanCache = {QueryPlanCache@15258}
    cacheAccess = {CacheImpl@15259}
    isClosed = false
    typeResolver = {TypeResolver@15260}
    typeHelper = {TypeLocatorImpl@15261}
    transactionEnvironment = {TransactionEnvironmentImpl@15262}
    sessionFactoryOptions = {SessionFactoryImpl$1@15263}
    customEntityDirtinessStrategy = {SessionFactoryImpl$3@15264}
    currentTenantIdentifierResolver = null
  propertyNames = {String[2]@15216}
  propertyTypes = {Type[2]@15217}
  propertyNullability = {boolean[2]@15218}
  propertySpan = 2
  cascade = {CascadeStyle[2]@15219}
  joinedFetch = {FetchMode[2]@15220}
  isKey = true
  hasNotNullProperty = true
  entityMode = {EntityMode@15221} "pojo"
  componentTuplizer = {PojoComponentTuplizer@15222}
  canDoExtraction = null
entityOrRoleName = "com.project.data.Account"
tenantId = null
hashCode = 203299394
key = {BrandIdPK@15175} "BrandIdPK[id=1,brand=BRAND]"
type = {ComponentType@15211}
entityOrRoleName = "com.project.data.Account"
tenantId = null
hashCode = 203299394



my assumption is that session factory is serialized differently on node1 and node2


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Wed Nov 01, 2017 5:01 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
The type objects are identical. Am I missing something?


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Wed Nov 01, 2017 5:26 am 
Newbie

Joined: Mon Oct 30, 2017 8:02 am
Posts: 5
yeah, right my bad, after i added hibernate.session_factory_name property and set hibernate.session_factory_name_is_jndi to false session factory looks the same. Now I am completely confused why key serialized differently


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Wed Nov 01, 2017 6:57 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
Maybe it's something else. Try adding logging to gather more details from the system.


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Thu Nov 02, 2017 7:15 am 
Newbie

Joined: Mon Oct 30, 2017 8:02 am
Posts: 5
i have found a workaround for this problem I added a custom serializer for CacheKey, and instead of serializing whole CacheKey object, use only key ref only. So the problem is in serialization of these fields:

private final Type type;
private final String entityOrRoleName;
private final String tenantId;
private final int hashCode;


Top
 Profile  
 
 Post subject: Re: Hibernate second level cache consistency
PostPosted: Thu Nov 02, 2017 8:11 am 
Hibernate Team
Hibernate Team

Joined: Thu Sep 11, 2014 2:50 am
Posts: 1628
Location: Romania
If you can replicate it with our template, you should open a new Jira issue.


Top
 Profile  
 
Display posts from previous:  Sort by  
Forum locked This topic is locked, you cannot edit posts or make further replies.  [ 10 posts ] 

All times are UTC - 5 hours [ DST ]


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
© Copyright 2014, Red Hat Inc. All rights reserved. JBoss and Hibernate are registered trademarks and servicemarks of Red Hat, Inc.