I tested your RegEx, and from what I see your RegEx only checks if the string is literally txt, jpeg, jpg, etc.
So that means that a string like 'jpg’ will return a match, and a string like 'filename.jpg’.
Like Tim already said, if you change the RegEx to this in Mendix (in case you're using the isMatch function) it should work: .*(txt$|jpeg$|png$|pdf$|jpg$)
The '.*’ at the beginning of the expression means that there has to be any character before the value of any element in your list of possible file formats.
Are you sure that the file extension of the file that you are checking doesn't contain a capital letter? For example Jpg or Jpeg?
Are you testing your filename against that regex in Java or in Mendix? If in Mendix and you are using IsMatch, take note that IsMatch prepends the regex with '^’ and postpends it with ‘$’ so you might lose the $ and add '.*’ in front.