openssl: Some strange errors with using lybcrypto-3.dll externally
Hello! I am a Lua programmer and i need to use some of openssl cryptographic functions in Lua, such as HMAC SHA-512, so i decide to use it by the writing Dynamic Link Library in C with exported function that communicates with OpenSSL modules - they computing hash and returning it to the DLL and the DLL return this hash to the Lua code, So there is my code of DLL in C on the MS Visual Studio 2017:
#define LUA_LIB
#define LUA_BUILD_AS_DLL
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"
#include "luaconf.h"
#include <openssl/hmac.h>
#include "hmac_lcl.h"
#include "stdafx.h"
static int forLua_Hmacsha512(lua_State *L) {
size_t lSecret, lMsg;
const char* secret = luaL_checklstring(L, 1, &lSecret);
const unsigned char* msg = (const unsigned char*)luaL_checklstring(L, 2, &lMsg);
HMAC_CTX ctx;
HMAC_CTX_reset(&ctx);
HMAC_Init_ex(&ctx, secret, lSecret, EVP_sha512(), NULL);
HMAC_Update(&ctx, msg, lMsg);
unsigned char result[129];
unsigned int result_len = 129;
HMAC_Final(&ctx, result, &result_len);
lua_pushlstring(L, (const char*)result, result_len);
return 1;
}
LUALIB_API int __declspec(dllexport) luaopen_Hm512(lua_State* L) {
lua_newtable(L);
lua_pushstring(L, "Hm512");
lua_pushcfunction(L, forLua_Hmacsha512);
lua_rawset(L, -3);
return 1;
}
this code need to include in Additional Dependency of Linker of properties of the project
the previously compiled into static library the Hmac sources from \openssl-master\crypto\hmac
and Lua51.lib ( or Lua53.lib or othe version of Lua)
After i compiled my project into DLL naming it for example Hm512 - Hm512.dll i tried to use it in lua script to get Hmac Sha-512 hash, for this it needs to put libcrypto-3.dll and compiled Hmac512.dll in the folder of Lua interpreter binaries Lua script looks like this:
hm = require ("Hm512")
x=hm.Hm512("sfgds","drgsd")
after trying to execute this script it always gives me error and Windows suggest me an option - stops the program Lua.exe or debug it, if i choose to debug it with Visual Studio i see this: https://cloud.mail.ru/public/EudE%2FSr82auPUz https://cloud.mail.ru/public/G9MM%2FLxkm1Jbq3 So the C code stops on the error in string -
if (!EVP_MD_CTX_test_flags(ctx, EVP_MD_CTX_FLAG_KEEP_PKEY_CTX))
Unhandled exception thrown: read acces violation. ctx was 0xCCCCCCCC


by calling function EVP_MD_CTX_test_flags from the \openssl-master\crypto\evp\digest.c line 910:
int EVP_MD_CTX_test_flags(const EVP_MD_CTX *ctx, int flags)
So i tried to build my Hm512.dll library with previous versions of OpenSSL and its libcrypto.dll, of openssl - 1.1.1d, 1.1.0l and others, and to execute my Lua script with newly builded Hm512.dll library on its sources and with libcrypto.dll of its version openssl, and libcrypto.dll of all version of OpenSSL gives me the same error:
if (!EVP_MD_CTX_test_flags(ctx, EVP_MD_CTX_FLAG_KEEP_PKEY_CTX))
Unhandled exception thrown: read acces violation. ctx was 0xCCCCCCCC
by calling function EVP_MD_CTX_test_flags from the \openssl-master\crypto\evp\digest.c line :
int EVP_MD_CTX_test_flags(const EVP_MD_CTX *ctx, int flags)
How it may be that your library throwns error if try to use it externally?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (12 by maintainers)
I’ve corrected my code with your suggestion above and now it works! I am very grateful for your help! Thank you very much!
Don’t do this! The *_lcl.h files are internal only and are not intended for external application use. If you dive under the covers and start using these files directly then all bets are off…we really can’t support you.
As @levitte said above:
So your code needs to look something like this (error checking omitted for brevity):
The function
HMAC_CTX_reset()does what its name says, it resets an already initialized context by callinghmac_ctx_cleanup(). But the provided context is not initialized.https://github.com/openssl/openssl/blob/0a4d6c67480a4d2fce514e08d3efe571f2ee99c9/crypto/hmac/hmac.c#L184-L192
That’s where the uninitialized pointers get dereferenced:
https://github.com/openssl/openssl/blob/0a4d6c67480a4d2fce514e08d3efe571f2ee99c9/crypto/hmac/hmac.c#L146-L154
Or maybe it’s just that the stack allocated
HMAC_CTXstructure is filled entirely with the 0xCC pattern by the compiler, because it does not get initialized explicitly, whencemd_ctxandi_ctx, ando_ctxare filled with the 0xCC pattern, too.One of those pointers then get passed to
EVP_MD_CTX_test_flags(), causing the crash. Does that make sense to you?